Interpreting the decisions of complex computer vision models is crucial to establish trust and accountability, especially in safety-critical domains. An established approach to interpretability is generating visual attribution maps that highlight regions of the input most relevant to the model's prediction. However, existing methods face a three-way trade-off. Propagation-based approaches are efficient, but they can be biased and architecture-specific. Meanwhile, perturbation-based methods are causally grounded, yet they are expensive and for vision transformers often yield coarse, patch-level explanations. Learning-based explainers are fast but usually optimize surrogate objectives or distill from heuristic teachers. We propose a learning scheme that instead optimizes deletion and insertion metrics directly. Since these metrics depend on non-differentiable sorting and ranking, we frame them as permutation learning and replace the hard sorting with a differentiable relaxation using Gumbel-Sinkhorn. This enables end-to-end training through attribution-guided perturbations of the target model. During inference, our method produces dense, pixel-level attributions in a single forward pass with optional, few-step gradient refinement. Our experiments demonstrate consistent quantitative improvements and sharper, boundary-aligned explanations, particularly for transformer-based vision models.

翻译：解释复杂计算机视觉模型的决策对于建立信任和问责制至关重要，尤其是在安全关键领域。一种成熟的解释性方法是生成视觉归因图，突出显示与模型预测最相关的输入区域。然而，现有方法面临三重权衡：基于传播的方法效率高，但可能存在偏差且依赖于特定架构；基于扰动的方法具有因果基础，但计算成本高，且对于视觉Transformer通常只能生成粗糙的块级解释；基于学习的解释器速度快，但通常优化替代目标或从启发式教师模型中提取知识。我们提出一种学习方案，直接优化删除和插入指标。由于这些指标依赖于不可微分的排序和排名，我们将其构建为排列学习问题，并使用Gumbel-Sinkhorn算子将硬排序替换为可微松弛。这使得目标模型能够通过归因引导的扰动进行端到端训练。在推理阶段，我们的方法通过单次前向传播生成密集的像素级归因，并可选用少量步骤的梯度细化。实验表明，我们的方法在定量指标上表现一致更优，且能产生更清晰、与边界对齐的解释，尤其适用于基于Transformer的视觉模型。