Vision Transformers (ViTs) have achieved state-of-the-art performance for various vision tasks. One reason behind the success lies in their ability to provide plausible innate explanations for the behavior of neural architectures. However, ViTs suffer from issues with explanation faithfulness, as their focal points are fragile to adversarial attacks and can be easily changed with even slight perturbations on the input image. In this paper, we propose a rigorous approach to mitigate these issues by introducing Faithful ViTs (FViTs). Briefly speaking, an FViT should have the following two properties: (1) The top-$k$ indices of its self-attention vector should remain mostly unchanged under input perturbation, indicating stable explanations; (2) The prediction distribution should be robust to perturbations. To achieve this, we propose a new method called Denoised Diffusion Smoothing (DDS), which adopts randomized smoothing and diffusion-based denoising. We theoretically prove that processing ViTs directly with DDS can turn them into FViTs. We also show that Gaussian noise is nearly optimal for both $\ell_2$ and $\ell_\infty$-norm cases. Finally, we demonstrate the effectiveness of our approach through comprehensive experiments and evaluations. Results show that FViTs are more robust against adversarial attacks while maintaining the explainability of attention, indicating higher faithfulness.

翻译：视觉变换器（ViTs）已在多种视觉任务上取得最先进性能。其成功原因之一在于能为神经架构的行为提供看似合理的固有解释。然而，ViT在解释忠实度方面存在问题，因为其关注点容易受到对抗攻击的干扰，即使输入图像发生轻微扰动，焦点也可能轻易改变。本文提出了一种严谨方法来解决这些问题，通过引入忠实ViT（FViT）。简言之，FViT应具备以下两个特性：（1）其自注意力向量的前$k$个索引在输入扰动下应基本保持不变，表明解释具有稳定性；（2）预测分布应对扰动具有鲁棒性。为此，我们提出了一种名为去噪扩散平滑（DDS）的新方法，该方法结合了随机平滑和基于扩散的去噪。我们从理论上证明，直接用DDS处理ViT可将其转化为FViT。同时表明，对于$\ell_2$和$\ell_\infty$范数情形，高斯噪声几乎是最优选择。最后，通过全面的实验和评估验证了该方法的有效性。结果表明，FViT在保持注意力可解释性的同时，对对抗攻击具有更强的鲁棒性，体现出更高的忠实度。