Self-attention has become a defacto choice for capturing global context in various vision applications. However, its quadratic computational complexity with respect to image resolution limits its use in real-time applications, especially for deployment on resource-constrained mobile devices. Although hybrid approaches have been proposed to combine the advantages of convolutions and self-attention for a better speed-accuracy trade-off, the expensive matrix multiplication operations in self-attention remain a bottleneck. In this work, we introduce a novel efficient additive attention mechanism that effectively replaces the quadratic matrix multiplication operations with linear element-wise multiplications. Our design shows that the key-value interaction can be replaced with a linear layer without sacrificing any accuracy. Unlike previous state-of-the-art methods, our efficient formulation of self-attention enables its usage at all stages of the network. Using our proposed efficient additive attention, we build a series of models called "SwiftFormer" which achieves state-of-the-art performance in terms of both accuracy and mobile inference speed. Our small variant achieves 78.5% top-1 ImageNet-1K accuracy with only 0.8 ms latency on iPhone 14, which is more accurate and 2x faster compared to MobileViT-v2. Code: https://github.com/Amshaker/SwiftFormer

翻译：自注意力机制已成为各类视觉应用中捕获全局上下文的事实标准。然而，其计算复杂度随图像分辨率呈二次增长，限制了它在实时应用中的使用，尤其是在资源受限的移动设备部署场景中。尽管已有混合方法试图结合卷积与自注意力的优势以实现速度与精度的更好平衡，但自注意力中昂贵的矩阵乘法运算仍是瓶颈。本文提出一种新型高效加性注意力机制，该机制将二次矩阵乘法运算有效替换为线性逐元素乘法。我们的设计表明，键值交互可通过线性层替代而不损失任何精度。与现有最先进方法不同，我们的高效自注意力公式可应用于网络的所有阶段。基于所提出的高效加性注意力，我们构建了一系列名为"SwiftFormer"的模型，其在精度和移动端推理速度方面均达到最优性能。小型变体在iPhone 14上仅需0.8毫秒延迟即可实现78.5%的ImageNet-1K top-1准确率，与MobileViT-v2相比精度更高且速度快2倍。代码：https://github.com/Amshaker/SwiftFormer