Recently, Transformers have shown promising performance in various vision tasks. However, the high costs of global self-attention remain challenging for Transformers, especially for high-resolution vision tasks. Inspired by one of the most successful transformers-based models for NLP: Big Bird, we propose a novel sparse attention mechanism for Vision Transformers (ViT). Specifically, we separate the heads into three groups, the first group used convolutional neural network (CNN) to extract local features and provide positional information for the model, the second group used Random Sampling Windows (RS-Win) for sparse self-attention calculation, and the third group reduces the resolution of the keys and values by average pooling for global attention. Based on these components, ViT maintains the sparsity of self-attention while maintaining the merits of Big Bird (i.e., the model is a universal approximator of sequence functions and is Turing complete). Moreover, our results show that the positional encoding, a crucial component in ViTs, can be safely removed in our model. Experiments show that Vision Big Bird demonstrates competitive performance on common vision tasks.

翻译：近期，Transformer在各类视觉任务中展现出优异性能。然而，全局自注意力机制的高昂计算成本仍对Transformer构成挑战，尤其在高分辨率视觉任务中。受NLP领域最成功的Transformer模型之一Big Bird启发，我们提出一种面向视觉Transformer（ViT）的新型稀疏注意力机制。具体而言，我们将注意力头划分为三组：第一组利用卷积神经网络（CNN）提取局部特征并为模型提供位置信息；第二组采用随机采样窗口（RS-Win）进行稀疏自注意力计算；第三组通过平均池化降低键和值的分辨率以实现全局注意力。基于这些组件，ViT在保持自注意力稀疏性的同时，继承了Big Bird的优势（即模型是序列函数的通用逼近器且具有图灵完备性）。此外，我们的结果表明，视觉Transformer中的关键组件位置编码可在本模型中安全移除。实验证明，Vision Big Bird在常见视觉任务中展现出具有竞争力的性能。