The success of Vision Transformer (ViT) has been widely reported on a wide range of image recognition tasks. ViT can learn global dependencies superior to CNN, yet CNN's inherent locality can substitute for expensive training resources. Recently, the outstanding performance of RetNet in the field of language modeling has garnered attention, surpassing that of the Transformer with explicit local modeling, shifting researchers' focus towards Transformers in the CV field. This paper investigates the effectiveness of RetNet from a CNN perspective and presents a variant of RetNet tailored to the visual domain. Similar to RetNet we improves ViT's local modeling by applying a weight mask on the original self-attention matrix. A straightforward way to locally adapt the self-attention matrix can be realized by an element-wise learnable weight mask (ELM), for which our preliminary results show promising results. However, the element-wise simple learnable weight mask not only induces a non-trivial additional parameter overhead but also increases the optimization complexity. To this end, this work proposes a novel Gaussian mixture mask (GMM) in which one mask only has two learnable parameters and it can be conveniently used in any ViT variants whose attention mechanism allows the use of masks. Experimental results on multiple small datasets demonstrate that the effectiveness of our proposed Gaussian mask for boosting ViTs for free (almost zero additional parameter or computation cost). Our code can be publicly available at https://github.com/CatworldLee/Gaussian-Mixture-Mask-Attention.

翻译：视觉Transformer（ViT）的成功已在广泛的图像识别任务中得到广泛报道。ViT能够学习优于CNN的全局依赖关系，而CNN固有的局部性则可以替代昂贵的训练资源。近年来，RetNet在语言建模领域展现出的卓越性能引起了关注，其表现超越了具有显式局部建模的Transformer，促使研究者将目光转向计算机视觉领域的Transformer。本文从CNN视角探究RetNet的有效性，并提出一种适用于视觉领域的RetNet变体。与RetNet类似，我们通过对原始自注意力矩阵施加权重掩码来增强ViT的局部建模能力。一种直接的局部化自注意力矩阵的方法可通过逐元素可学习权重掩码（ELM）实现，初步结果显示该方法颇具前景。然而，这种逐元素的简单可学习权重掩码不仅带来了不可忽视的额外参数开销，还增加了优化复杂度。为此，本文提出了一种新颖的高斯混合掩码（GMM），其中每个掩码仅包含两个可学习参数，并可便捷地应用于任何允许使用掩码的自注意力机制的ViT变体。在多个小规模数据集上的实验结果表明，我们提出的高斯掩码能够以近乎为零的额外参数或计算成本有效提升ViT性能。我们的代码已在https://github.com/CatworldLee/Gaussian-Mixture-Mask-Attention开源。