This paper presents a new vision Transformer, Scale-Aware Modulation Transformer (SMT), that can handle various downstream tasks efficiently by combining the convolutional network and vision Transformer. The proposed Scale-Aware Modulation (SAM) in the SMT includes two primary novel designs. Firstly, we introduce the Multi-Head Mixed Convolution (MHMC) module, which can capture multi-scale features and expand the receptive field. Secondly, we propose the Scale-Aware Aggregation (SAA) module, which is lightweight but effective, enabling information fusion across different heads. By leveraging these two modules, convolutional modulation is further enhanced. Furthermore, in contrast to prior works that utilized modulations throughout all stages to build an attention-free network, we propose an Evolutionary Hybrid Network (EHN), which can effectively simulate the shift from capturing local to global dependencies as the network becomes deeper, resulting in superior performance. Extensive experiments demonstrate that SMT significantly outperforms existing state-of-the-art models across a wide range of visual tasks. Specifically, SMT with 11.5M / 2.4GFLOPs and 32M / 7.7GFLOPs can achieve 82.2% and 84.3% top-1 accuracy on ImageNet-1K, respectively. After pretrained on ImageNet-22K in 224^2 resolution, it attains 87.1% and 88.1% top-1 accuracy when finetuned with resolution 224^2 and 384^2, respectively. For object detection with Mask R-CNN, the SMT base trained with 1x and 3x schedule outperforms the Swin Transformer counterpart by 4.2 and 1.3 mAP on COCO, respectively. For semantic segmentation with UPerNet, the SMT base test at single- and multi-scale surpasses Swin by 2.0 and 1.1 mIoU respectively on the ADE20K.

翻译：本文提出了一种新的视觉Transformer——尺度感知调制Transformer（SMT），通过融合卷积网络与视觉Transformer，高效处理各类下游任务。SMT中的核心模块——尺度感知调制（SAM）包含两项关键创新设计：首先，我们引入多头混合卷积（MHMC）模块，该模块能够捕获多尺度特征并扩展感受野。其次，我们提出轻量高效的尺度感知聚合（SAA）模块，实现不同头部间的信息融合。通过这两个模块的协同作用，卷积调制能力得到进一步增强。与以往在全阶段使用调制构建免注意力网络的工作不同，我们提出进化式混合网络（EHN），该方法能有效模拟网络深度增加时从局部依赖到全局依赖的转移过程，从而获得更优性能。大量实验表明，SMT在多种视觉任务中显著超越现有最优模型。具体而言，参数量11.5M/计算量2.4GFLOPs和参数量32M/计算量7.7GFLOPs的SMT模型在ImageNet-1K上分别达到82.2%和84.3%的Top-1准确率；在224^2分辨率下经ImageNet-22K预训练后，以224^2和384^2分辨率微调时分别取得87.1%和88.1%的Top-1准确率；在目标检测任务中，使用Mask R-CNN的SMT基准模型在1x和3x训练计划下，在COCO数据集上分别比Swin Transformer提升4.2和1.3个mAP；在语义分割任务中，使用UPerNet的SMT基准模型在ADE20K数据集上单尺度和多尺度测试分别比Swin模型提升2.0和1.1个mIoU。