Explainable artificial intelligence (XAI) methods shed light on the predictions of machine learning algorithms. Several different approaches exist and have already been applied in climate science. However, usually missing ground truth explanations complicate their evaluation and comparison, subsequently impeding the choice of the XAI method. Therefore, in this work, we introduce XAI evaluation in the climate context and discuss different desired explanation properties, namely robustness, faithfulness, randomization, complexity, and localization. To this end, we chose previous work as a case study where the decade of annual-mean temperature maps is predicted. After training both a multi-layer perceptron (MLP) and a convolutional neural network (CNN), multiple XAI methods are applied and their skill scores in reference to a random uniform explanation are calculated for each property. Independent of the network, we find that XAI methods Integrated Gradients, layer-wise relevance propagation, and input times gradients exhibit considerable robustness, faithfulness, and complexity while sacrificing randomization performance. Sensitivity methods -- gradient, SmoothGrad, NoiseGrad, and FusionGrad, match the robustness skill but sacrifice faithfulness and complexity for randomization skill. We find architecture-dependent performance differences regarding robustness, complexity and localization skills of different XAI methods, highlighting the necessity for research task-specific evaluation. Overall, our work offers an overview of different evaluation properties in the climate science context and shows how to compare and benchmark different explanation methods, assessing their suitability based on strengths and weaknesses, for the specific research problem at hand. By that, we aim to support climate researchers in the selection of a suitable XAI method.

翻译：可解释人工智能（XAI）方法能够揭示机器学习算法的预测依据。目前已有多种不同方法应用于气候科学领域，但解释结果通常缺乏真实基准（ground truth），这导致对其评估与比较存在困难，进而阻碍了XAI方法的选择。为此，本研究首次引入气候背景下的XAI评估框架，探讨了五种期望的解释属性：鲁棒性、保真性、随机化、复杂性与局部特性。我们以预测年均温十年尺度分布图的现有研究作为案例，在训练多层感知器（MLP）和卷积神经网络（CNN）后，应用多种XAI方法，并计算其相对于随机均匀解释的技能得分。研究发现，不论网络结构如何，Integrated Gradients、逐层相关性传播（layer-wise relevance propagation）及输入×梯度（input times gradients）方法均表现出显著的鲁棒性、保真性和复杂性，但随机化性能有所牺牲；而敏感性方法（包括梯度、SmoothGrad、NoiseGrad和FusionGrad）虽能匹配鲁棒性技能，却以牺牲保真性和复杂性为代价获得随机化优势。不同XAI方法在鲁棒性、复杂性和局部特性方面存在架构依赖性的性能差异，凸显了针对具体研究任务进行专项评估的必要性。总体而言，本研究系统梳理了气候科学领域的评估属性体系，展示了如何对不同解释方法进行比较与基准测试，依据其优劣势为具体研究问题选择合适方法，旨在帮助气候研究人员筛选适用的XAI方法。