While models derived from Vision Transformers (ViTs) have been phonemically surging, pre-trained models cannot seamlessly adapt to arbitrary resolution images without altering the architecture and configuration, such as sampling the positional encoding, limiting their flexibility for various vision tasks. For instance, the Segment Anything Model (SAM) based on ViT-Huge requires all input images to be resized to 1024$\times$1024. To overcome this limitation, we propose the Multi-Head Self-Attention Convolution (MSA-Conv) that incorporates Self-Attention within generalized convolutions, including standard, dilated, and depthwise ones. Enabling transformers to handle images of varying sizes without retraining or rescaling, the use of MSA-Conv further reduces computational costs compared to global attention in ViT, which grows costly as image size increases. Later, we present the Vision Transformer in Convolution (TiC) as a proof of concept for image classification with MSA-Conv, where two capacity enhancing strategies, namely Multi-Directional Cyclic Shifted Mechanism and Inter-Pooling Mechanism, have been proposed, through establishing long-distance connections between tokens and enlarging the effective receptive field. Extensive experiments have been carried out to validate the overall effectiveness of TiC. Additionally, ablation studies confirm the performance improvement made by MSA-Conv and the two capacity enhancing strategies separately. Note that our proposal aims at studying an alternative to the global attention used in ViT, while MSA-Conv meets our goal by making TiC comparable to state-of-the-art on ImageNet-1K. Code will be released at https://github.com/zs670980918/MSA-Conv.

翻译：尽管源自视觉变换器（ViT）的模型取得了显著进展，但预训练模型无法在不改变架构和配置（如对位置编码进行采样）的情况下无缝适应任意分辨率图像，这限制了其在不同视觉任务中的灵活性。例如，基于ViT-Huge的“分割一切模型”（SAM）要求将所有输入图像调整为1024×1024。为克服这一局限，我们提出了多头自注意力卷积（MSA-Conv），它将自注意力融入广义卷积中，包括标准卷积、膨胀卷积和深度可分离卷积。使用MSA-Conv使得变换器能够在不重新训练或缩放的情况下处理不同尺寸的图像，同时相比ViT中随图像尺寸增长而计算成本激增的全局注意力，进一步降低了计算开销。随后，我们提出了卷积中的视觉变换器（TiC）作为MSA-Conv在图像分类中的概念验证，其中引入了两种容量增强策略——多方向循环移位机制和池间机制——通过建立token间的长距离连接和扩大有效感受野来实现。大量实验验证了TiC的整体有效性。此外，消融研究分别确认了MSA-Conv和两种容量增强策略带来的性能提升。需指出，本工作旨在探索ViT中全局注意力的替代方案，而MSA-Conv通过使TiC在ImageNet-1K上达到与最先进方法可比的性能，实现了我们的目标。代码将发布在https://github.com/zs670980918/MSA-Conv。