Large Language Models (LLMs) offer natural language explanations as an alternative to feature attribution methods for model interpretability. However, despite their plausibility, they may not reflect the model's true reasoning faithfully. While several faithfulness metrics have been proposed, they are often evaluated in isolation, making principled comparisons between them difficult. We present Causal Diagnosticity, a testbed framework for evaluating faithfulness metrics for natural language explanations. We use the concept of diagnosticity, and employ model-editing methods to generate faithful-unfaithful explanation pairs. Our benchmark includes four tasks: fact-checking, analogy, object counting, and multi-hop reasoning. We evaluate prominent faithfulness metrics, including post-hoc explanation and chain-of-thought methods. Diagnostic performance varies across tasks and models, with Filler Tokens performing best overall. Additionally, continuous metrics are generally more diagnostic than binary ones but can be sensitive to noise and model choice. Our results highlight the need for more robust faithfulness metrics.

翻译：大型语言模型（LLMs）为模型可解释性提供了自然语言解释作为特征归因方法的替代方案。然而，尽管这些解释看似合理，却可能无法忠实反映模型的真实推理过程。尽管已有多种忠实度评估指标被提出，但它们往往被孤立评估，导致难以进行原则性的比较。我们提出“因果诊断性”这一测试基准框架，用于评估自然语言解释的忠实度指标。我们运用诊断性概念，并采用模型编辑方法生成忠实与非忠实解释对。我们的基准测试包含四项任务：事实核查、类比推理、物体计数和多跳推理。我们评估了包括事后解释和思维链方法在内的主流忠实度指标。诊断性能在不同任务和模型间存在差异，其中填充词标记方法整体表现最佳。此外，连续型指标通常比二元指标更具诊断性，但对噪声和模型选择较为敏感。我们的研究结果凸显了对更鲁棒的忠实度评估指标的迫切需求。