Vision Transformer (ViT) has shown great potential for various visual tasks due to its ability to model long-range dependency. However, ViT requires a large amount of computing resource to compute the global self-attention. In this work, we propose a ladder self-attention block with multiple branches and a progressive shift mechanism to develop a light-weight transformer backbone that requires less computing resources (e.g. a relatively small number of parameters and FLOPs), termed Progressive Shift Ladder Transformer (PSLT). First, the ladder self-attention block reduces the computational cost by modelling local self-attention in each branch. In the meanwhile, the progressive shift mechanism is proposed to enlarge the receptive field in the ladder self-attention block by modelling diverse local self-attention for each branch and interacting among these branches. Second, the input feature of the ladder self-attention block is split equally along the channel dimension for each branch, which considerably reduces the computational cost in the ladder self-attention block (with nearly 1/3 the amount of parameters and FLOPs), and the outputs of these branches are then collaborated by a pixel-adaptive fusion. Therefore, the ladder self-attention block with a relatively small number of parameters and FLOPs is capable of modelling long-range interactions. Based on the ladder self-attention block, PSLT performs well on several vision tasks, including image classification, objection detection and person re-identification. On the ImageNet-1k dataset, PSLT achieves a top-1 accuracy of 79.9% with 9.2M parameters and 1.9G FLOPs, which is comparable to several existing models with more than 20M parameters and 4G FLOPs. Code is available at https://isee-ai.cn/wugaojie/PSLT.html.

翻译：视觉Transformer（ViT）因具备建模长距离依赖的能力，在各类视觉任务中展现出巨大潜力。然而，ViT需要大量计算资源以计算全局自注意力。本文提出了一种包含多分支的梯形自注意力模块及渐进位移机制，用于构建轻量级Transformer主干网络（称为渐进位移梯形Transformer，简称PSLT），该网络所需计算资源较少（例如参数量和FLOPs相对较小）。首先，梯形自注意力模块通过在各分支内建模局部自注意力来降低计算成本。同时，提出渐进位移机制，通过为每个分支建模不同的局部自注意力并实现分支间交互，以扩大梯形自注意力模块的感受野。其次，梯形自注意力模块的输入特征沿通道维度等分为各分支的输入，这显著降低了梯形自注意力模块的计算成本（参数量和FLOPs降至约1/3），随后通过像素自适应融合整合各分支输出。因此，参数量和FLOPs较小的梯形自注意力模块能够建模长距离交互。基于该梯形自注意力模块，PSLT在图像分类、目标检测和行人重识别等多项视觉任务中表现优异。在ImageNet-1k数据集上，PSLT以9.2M参数和1.9G FLOPs实现了79.9%的Top-1准确率，其性能可与参数超过20M、FLOPs超过4G的现有若干模型相媲美。代码已开源在https://isee-ai.cn/wugaojie/PSLT.html。