Explainability has become a central requirement for the development, deployment, and adoption of machine learning (ML) models and we are yet to understand what explanation methods can and cannot do. Several factors such as data, model prediction, hyperparameters used in training the model, and random initialization can all influence downstream explanations. While previous work empirically hinted that explanations (E) may have little relationship with the prediction (Y), there is a lack of conclusive study to quantify this relationship. Our work borrows tools from causal inference to systematically assay this relationship. More specifically, we measure the relationship between E and Y by measuring the treatment effect when intervening on their causal ancestors (hyperparameters) (inputs to generate saliency-based Es or Ys). We discover that Y's relative direct influence on E follows an odd pattern; the influence is higher in the lowest-performing models than in mid-performing models, and it then decreases in the top-performing models. We believe our work is a promising first step towards providing better guidance for practitioners who can make more informed decisions in utilizing these explanations by knowing what factors are at play and how they relate to their end task.

翻译：可解释性已成为机器学习（ML）模型开发、部署和应用的核心要求，然而我们尚未完全理解解释方法的能力边界。数据、模型预测、训练超参数和随机初始化等因素均可能影响下游解释结果。尽管先前研究通过经验证据暗示解释（E）与预测（Y）之间可能存在微弱关联，但缺乏定量分析该关系的确定性研究。本文借鉴因果推理工具系统分析了这一关系。具体而言，我们通过干预解释和预测的因果祖先（超参数）（即生成基于显著性的E或Y的输入）来测量治疗效应，以量化E与Y之间的关联。研究发现，Y对E的相对直接影响呈现反常模式：在低性能模型中这种影响最强，中等性能模型中减弱，而高性能模型中进一步降低。我们认为本研究是重要开端，有望通过揭示影响因素及其与最终任务的关系，为从业者更明智地运用解释方法提供有效指导。