Despite the success of vision transformers (ViTs), they still suffer from significant drops in accuracy in the presence of common corruptions, such as noise or blur. Interestingly, we observe that the attention mechanism of ViTs tends to rely on few important tokens, a phenomenon we call token overfocusing. More critically, these tokens are not robust to corruptions, often leading to highly diverging attention patterns. In this paper, we intend to alleviate this overfocusing issue and make attention more stable through two general techniques: First, our Token-aware Average Pooling (TAP) module encourages the local neighborhood of each token to take part in the attention mechanism. Specifically, TAP learns average pooling schemes for each token such that the information of potentially important tokens in the neighborhood can adaptively be taken into account. Second, we force the output tokens to aggregate information from a diverse set of input tokens rather than focusing on just a few by using our Attention Diversification Loss (ADL). We achieve this by penalizing high cosine similarity between the attention vectors of different tokens. In experiments, we apply our methods to a wide range of transformer architectures and improve robustness significantly. For example, we improve corruption robustness on ImageNet-C by 2.4% while simultaneously improving accuracy by 0.4% based on state-of-the-art robust architecture FAN. Also, when finetuning on semantic segmentation tasks, we improve robustness on CityScapes-C by 2.4% and ACDC by 3.1%.

翻译：尽管视觉Transformer（ViTs）取得了成功，但在噪声或模糊等常见干扰下，其准确率仍会出现显著下降。有趣的是，我们观察到ViTs的注意力机制倾向于依赖少数重要令牌，这一现象我们称为令牌过度聚焦。更关键的是，这些令牌对干扰不具备鲁棒性，常常导致注意力模式高度发散。本文旨在通过两种通用技术缓解这一过度聚焦问题，使注意力更加稳定：第一，我们的令牌感知平均池化（TAP）模块鼓励每个令牌的局部邻域参与注意力机制。具体而言，TAP为每个令牌学习平均池化方案，从而能够自适应地考虑邻域中潜在重要令牌的信息。第二，我们通过注意力多样化损失（ADL）强制输出令牌从多样化的输入令牌集合中聚合信息，而非仅聚焦于少数令牌。具体实现是通过惩罚不同令牌注意力向量之间的高余弦相似度。实验中，我们将方法应用于多种Transformer架构，显著提升了鲁棒性。例如，基于最先进的鲁棒架构FAN，我们在ImageNet-C上提升了2.4%的干扰鲁棒性，同时准确率提升0.4%。此外，在语义分割任务微调时，我们在CityScapes-C上提升了2.4%的鲁棒性，在ACDC上提升了3.1%。