Despite a sea of interpretability methods that can produce plausible explanations, the field has also empirically seen many failure cases of such methods. In light of these results, it remains unclear for practitioners how to use these methods and choose between them in a principled way. In this paper, we show that for moderately rich model classes (easily satisfied by neural networks), any feature attribution method that is complete and linear -- for example, Integrated Gradients and SHAP -- can provably fail to improve on random guessing for inferring model behaviour. Our results apply to common end-tasks such as characterizing local model behaviour, identifying spurious features, and algorithmic recourse. One takeaway from our work is the importance of concretely defining end-tasks: once such an end-task is defined, a simple and direct approach of repeated model evaluations can outperform many other complex feature attribution methods.

翻译：尽管大量可解释性方法能够生成看似合理的解释，该领域也实证性地观察到这类方法存在诸多失效案例。鉴于这些结果，从业者仍不清楚如何以原则性方式使用这些方法并加以选择。本文证明，对于适度丰富的模型类别（神经网络可轻易满足），任何完备且线性的特征归因方法（例如Integrated Gradients和SHAP）在推断模型行为方面必然无法优于随机猜测。我们的结论适用于常见终端任务，包括刻画局部模型行为、识别虚假特征及算法反事实解释。本研究的一个重要启示在于明确定义终端任务的重要性：一旦此类终端任务被定义，通过重复模型评估的简单直接方法即可超越许多复杂的特征归因方法。