Shift equivariance is a fundamental principle that governs how we perceive the world - our recognition of an object remains invariant with respect to shifts. Transformers have gained immense popularity due to their effectiveness in both language and vision tasks. While the self-attention operator in vision transformers (ViT) is permutation-equivariant and thus shift-equivariant, patch embedding, positional encoding, and subsampled attention in ViT variants can disrupt this property, resulting in inconsistent predictions even under small shift perturbations. Although there is a growing trend in incorporating the inductive bias of convolutional neural networks (CNNs) into vision transformers, it does not fully address the issue. We propose an adaptive polyphase anchoring algorithm that can be seamlessly integrated into vision transformer models to ensure shift-equivariance in patch embedding and subsampled attention modules, such as window attention and global subsampled attention. Furthermore, we utilize depth-wise convolution to encode positional information. Our algorithms enable ViT, and its variants such as Twins to achieve 100% consistency with respect to input shift, demonstrate robustness to cropping, flipping, and affine transformations, and maintain consistent predictions even when the original models lose 20 percentage points on average when shifted by just a few pixels with Twins' accuracy dropping from 80.57% to 62.40%.

翻译：平移等变性是支配我们感知世界的基本原则——我们对物体的识别在平移变换下保持不变。Transformer因其在语言和视觉任务中的有效性而广受欢迎。虽然视觉Transformer中自注意力算子具有置换等变性，从而也具备平移等变性，但视觉Transformer变体中的分块嵌入、位置编码和子采样注意力可能破坏这一性质，导致即使在微小平移扰动下也会产生不一致的预测。尽管将卷积神经网络的归纳偏置融入视觉Transformer的趋势日益增长，但这并未完全解决问题。我们提出了一种自适应多相锚定算法，可无缝集成到视觉Transformer模型中，确保分块嵌入和子采样注意力模块（如窗口注意力和全局子采样注意力）的平移等变性。此外，我们利用深度可分离卷积编码位置信息。我们的算法使得ViT及其变体（如Twins）在输入平移下达到100%的一致性，对裁剪、翻转和仿射变换具有鲁棒性，并且即使在原始模型因仅平移几个像素而平均损失20个百分点（Twins的准确率从80.57%降至62.40%）的情况下，仍能保持一致的预测。