One of the unsolved challenges in the field of Explainable AI (XAI) is determining how to most reliably estimate the quality of an explanation method in the absence of ground truth explanation labels. Resolving this issue is of utmost importance as the evaluation outcomes generated by competing evaluation methods (or ''quality estimators''), which aim at measuring the same property of an explanation method, frequently present conflicting rankings. Such disagreements can be challenging for practitioners to interpret, thereby complicating their ability to select the best-performing explanation method. We address this problem through a meta-evaluation of different quality estimators in XAI, which we define as ''the process of evaluating the evaluation method''. Our novel framework, MetaQuantus, analyses two complementary performance characteristics of a quality estimator: its resilience to noise and reactivity to randomness, thus circumventing the need for ground truth labels. We demonstrate the effectiveness of our framework through a series of experiments, targeting various open questions in XAI such as the selection and hyperparameter optimisation of quality estimators. Our work is released under an open-source license (https://github.com/annahedstroem/MetaQuantus) to serve as a development tool for XAI- and Machine Learning (ML) practitioners to verify and benchmark newly constructed quality estimators in a given explainability context. With this work, we provide the community with clear and theoretically-grounded guidance for identifying reliable evaluation methods, thus facilitating reproducibility in the field.

翻译：可解释人工智能领域尚未解决的挑战之一是，在缺乏真值解释标签的情况下，如何最可靠地评估解释方法的质量。解决这一问题至关重要，因为旨在衡量解释方法同一属性的不同评估方法（或称为“质量评估器”）所生成的评估结果，往往会显示相互矛盾的排名。这种分歧可能使从业者难以解读，从而复杂化了他们选择最佳性能解释方法的能力。我们通过对可解释人工智能中不同质量评估器进行元评估来解决该问题，并将其定义为“评估评估方法的过程”。我们的新框架MetaQuantus分析了质量评估器的两个互补性能特征：其对噪声的鲁棒性以及对随机性的反应性，从而规避了对真值标签的需求。通过一系列针对可解释人工智能中各种开放性问题（如质量评估器的选择与超参数优化）的实验，我们展示了该框架的有效性。我们的工作以开源许可证发布（https://github.com/annahedstroem/MetaQuantus），旨在作为可解释人工智能和机器学习从业者的开发工具，用于在特定可解释性背景下验证和基准测试新构建的质量评估器。通过这项工作，我们为社区提供了清晰且有理论依据的指导，以识别可靠的评估方法，从而促进该领域的可重复性。