Vision Transformers (ViTs) have attracted a lot of popularity in recent years, due to their exceptional capabilities in modeling long-range spatial dependencies and scalability for large scale training. Although the training parallelism of self-attention mechanism plays an important role in retaining great performance, its quadratic complexity baffles the application of ViTs in many scenarios which demand fast inference. This effect is even more pronounced in applications in which autoregressive modeling of input features is required. In Natural Language Processing (NLP), a new stream of efforts has proposed parallelizable models with recurrent formulation that allows for efficient inference in generative applications. Inspired by this trend, we propose a new class of computer vision models, dubbed Vision Retention Networks (ViR), with dual parallel and recurrent formulations, which strike an optimal balance between fast inference and parallel training with competitive performance. In particular, ViR scales favorably for image throughput and memory consumption in tasks that require higher-resolution images due to its flexible formulation in processing large sequence lengths. The ViR is the first attempt to realize dual parallel and recurrent equivalency in a general vision backbone for recognition tasks. We have validated the effectiveness of ViR through extensive experiments with different dataset sizes and various image resolutions and achieved competitive performance. Code: https://github.com/NVlabs/ViR

翻译：视觉Transformer（ViTs）近年来因其在建模长程空间依赖性及大规模训练可扩展性方面的卓越能力而备受关注。尽管自注意力机制的训练并行性在保持优异性能方面起着重要作用，但其二次复杂度阻碍了ViTs在众多需要快速推理场景中的应用。在需要对输入特征进行自回归建模的应用中，这种影响更为显著。在自然语言处理（NLP）领域，一系列新工作提出了具有循环公式的可并行化模型，从而在生成式应用中实现高效推理。受此趋势启发，我们提出了一类新型计算机视觉模型——视觉保持网络（ViR），其具有并行与循环双重公式，在快速推理与并行训练之间取得了竞争性性能的最优平衡。特别地，由于在处理长序列时具有灵活的公式化设计，ViR在需要更高分辨率图像的任务中，在吞吐量和内存消耗方面展现出良好的可扩展性。ViR是首个在通用视觉骨干网络中实现并行与循环等价性的尝试，适用于识别任务。我们通过不同数据集规模和多种图像分辨率的大量实验验证了ViR的有效性，并取得了竞争性性能。代码：https://github.com/NVlabs/ViR