As a de facto solution, the vanilla Vision Transformers (ViTs) are encouraged to model long-range dependencies between arbitrary image patches while the global attended receptive field leads to quadratic computational cost. Another branch of Vision Transformers exploits local attention inspired by CNNs, which only models the interactions between patches in small neighborhoods. Although such a solution reduces the computational cost, it naturally suffers from small attended receptive fields, which may limit the performance. In this work, we explore effective Vision Transformers to pursue a preferable trade-off between the computational complexity and size of the attended receptive field. By analyzing the patch interaction of global attention in ViTs, we observe two key properties in the shallow layers, namely locality and sparsity, indicating the redundancy of global dependency modeling in shallow layers of ViTs. Accordingly, we propose Multi-Scale Dilated Attention (MSDA) to model local and sparse patch interaction within the sliding window. With a pyramid architecture, we construct a Multi-Scale Dilated Transformer (DilateFormer) by stacking MSDA blocks at low-level stages and global multi-head self-attention blocks at high-level stages. Our experiment results show that our DilateFormer achieves state-of-the-art performance on various vision tasks. On ImageNet-1K classification task, DilateFormer achieves comparable performance with 70% fewer FLOPs compared with existing state-of-the-art models. Our DilateFormer-Base achieves 85.6% top-1 accuracy on ImageNet-1K classification task, 53.5% box mAP/46.1% mask mAP on COCO object detection/instance segmentation task and 51.1% MS mIoU on ADE20K semantic segmentation task.

翻译：作为事实上的解决方案，原始视觉Transformer（ViTs）被鼓励建模任意图像块之间的长距离依赖关系，然而全局注意力感受野导致了二次计算成本。另一类视觉Transformer受CNN启发利用局部注意力，仅建模小邻域内图像块之间的交互。尽管这种方案降低了计算成本，但自然存在注意力感受野较小的问题，这可能限制其性能。在本工作中，我们探索有效的视觉Transformer以在计算复杂度与注意力感受野大小之间实现更优权衡。通过分析ViTs中全局注意力的图像块交互，我们观察到浅层存在两个关键特性——局部性与稀疏性，表明ViTs浅层中全局依赖建模的冗余性。据此，我们提出多尺度膨胀注意力（MSDA），在滑动窗口内建模局部且稀疏的图像块交互。通过金字塔架构，我们利用在低层级堆叠MSDA模块、高层级堆叠全局多头自注意力模块的方式构建多尺度膨胀Transformer（DilateFormer）。实验结果表明，我们的DilateFormer在各种视觉任务上均达到最先进性能。在ImageNet-1K分类任务中，DilateFormer以降低70%浮点运算量的代价取得了与现有最优模型相当的性能。我们的DilateFormer-Base在ImageNet-1K分类任务上达到85.6%的top-1准确率，在COCO目标检测/实例分割任务上分别取得53.5%的边界框mAP/46.1%的掩膜mAP，在ADE20K语义分割任务上取得51.1%的MS mIoU。