Recent works have demonstrated that transformer can achieve promising performance in computer vision, by exploiting the relationship among image patches with self-attention. While they only consider the attention in a single feature layer, but ignore the complementarity of attention in different levels. In this paper, we propose the broad attention to improve the performance by incorporating the attention relationship of different layers for vision transformer, which is called BViT. The broad attention is implemented by broad connection and parameter-free attention. Broad connection of each transformer layer promotes the transmission and integration of information for BViT. Without introducing additional trainable parameters, parameter-free attention jointly focuses on the already available attention information in different layers for extracting useful information and building their relationship. Experiments on image classification tasks demonstrate that BViT delivers state-of-the-art accuracy of 74.8\%/81.6\% top-1 accuracy on ImageNet with 5M/22M parameters. Moreover, we transfer BViT to downstream object recognition benchmarks to achieve 98.9\% and 89.9\% on CIFAR10 and CIFAR100 respectively that exceed ViT with fewer parameters. For the generalization test, the broad attention in Swin Transformer and T2T-ViT also bring an improvement of more than 1\%. To sum up, broad attention is promising to promote the performance of attention based models. Code and pre-trained models are available at https://github.com/DRL-CASIA/Broad_ViT.

翻译：近期研究表明，Transformer通过利用自注意力机制探索图像块间的关系，能够在计算机视觉领域取得优异性能。然而现有方法仅关注单一特征层的注意力，忽视了不同层级注意力的互补性。本文提出宽注意力机制（Broad Attention），通过整合视觉Transformer不同层的注意力关系来提升性能，由此提出BViT模型。该宽注意力通过宽连接（Broad Connection）与无参数注意力（Parameter-Free Attention）实现。各transformer层的宽连接促进了BViT的信息传递与融合。无参数注意力在无需引入额外可训练参数的情况下，协同聚焦不同层已有的注意力信息，以提取有效信息并构建其关联性。图像分类任务实验表明，BViT在ImageNet数据集上以5M/22M参数量分别达到74.8%/81.6%的Top-1准确率，取得当前最优水平。此外，我们将BViT迁移至下游目标识别基准测试，在CIFAR10和CIFAR100上分别获得98.9%和89.9%的准确率，以更少参数超越ViT。泛化测试表明，Swin Transformer与T2T-ViT中引入宽注意力机制后性能提升均超过1%。综上所述，宽注意力有望提升基于注意力模型的性能。代码与预训练模型已开源：https://github.com/DRL-CASIA/Broad_ViT。