This paper presents a novel approach to address the challenges of understanding the prediction process and debugging prediction errors in Vision Transformers (ViT), which have demonstrated superior performance in various computer vision tasks such as image classification and object detection. While several visual explainability techniques, such as CAM, Grad-CAM, Score-CAM, and Recipro-CAM, have been extensively researched for Convolutional Neural Networks (CNNs), limited research has been conducted on ViT. Current state-of-the-art solutions for ViT rely on class agnostic Attention-Rollout and Relevance techniques. In this work, we propose a new gradient-free visual explanation method for ViT, called ViT-ReciproCAM, which does not require attention matrix and gradient information. ViT-ReciproCAM utilizes token masking and generated new layer outputs from the target layer's input to exploit the correlation between activated tokens and network predictions for target classes. Our proposed method outperforms the state-of-the-art Relevance method in the Average Drop-Coherence-Complexity (ADCC) metric by $4.58\%$ to $5.80\%$ and generates more localized saliency maps. Our experiments demonstrate the effectiveness of ViT-ReciproCAM and showcase its potential for understanding and debugging ViT models. Our proposed method provides an efficient and easy-to-implement alternative for generating visual explanations, without requiring attention and gradient information, which can be beneficial for various applications in the field of computer vision.

翻译：本文提出了一种新方法，以应对视觉Transformer（ViT）在理解预测过程与调试预测错误方面的挑战。ViT在图像分类、目标检测等多项计算机视觉任务中展现出优越性能。尽管CAM、Grad-CAM、Score-CAM及Recipro-CAM等多种视觉可解释性技术已在卷积神经网络（CNN）领域得到广泛研究，但针对ViT的此类研究尚显不足。当前ViT领域的最优解决方案依赖类别无关的注意力展开（Attention-Rollout）及相关性（Relevance）技术。本文提出一种面向ViT的新型无梯度视觉解释方法——ViT-ReciproCAM，该方法无需注意力矩阵与梯度信息。ViT-ReciproCAM通过令牌掩码（token masking）及目标层输入生成的新层输出，挖掘激活令牌与网络对目标类别预测之间的相关性。在平均下降-连贯性-复杂度（ADCC）指标上，我们提出的方法较当前最优的相关性方法提升4.58%至5.80%，并生成更具局部性的显著性图。实验结果表明，ViT-ReciproCAM在理解与调试ViT模型方面具有显著有效性。该方法无需注意力与梯度信息，为生成视觉解释提供了一种高效且易于实现的替代方案，有望推动计算机视觉领域的多样化应用。