Evaluating methods of explainable artificial intelligence (XAI) is challenging, because the fidelity of an explanation to the AI model does not necessarily go hand in hand with its interpretability for humans. For instance, when classifying images with Convolutional Neural Networks (CNN), XAI algorithms can explain which image areas had an impact on the CNN decision. However, it is unclear whether the areas that best reflect the CNN internal data processing will also make most sense to humans. Thus, the present study investigated whether the image classification of humans and CNN is supported by the same explanations. To assess such differences in interpretability, human participants and a CNN classified image segments that were considered most informative either by other humans (as revealed by eye movements and manual selection) or by two XAI methods (Grad-CAM and XRAI). In three experiments, humans classified and rated these segments, and we also had a CNN classify them. The results indicated that the respective interpretability of the two XAI methods strongly depended on image type, both for humans and CNN. Moreover, human classification performance was highest with human segments, regardless of how they were generated (i.e., from eye movements or manual selection), whereas the type of human segment had major impacts on CNN classification performance. Our results caution against general statements about the interpretability of explanations, as this interpretability varies with the explanation method, the explanations to be interpreted, and the agent who needs to perform the interpretation.

翻译：评估可解释人工智能（XAI）方法具有挑战性，因为对AI模型的解释忠实度并不必然与其对人类的可解释性保持一致。例如，在使用卷积神经网络（CNN）进行图像分类时，XAI算法可以解释哪些图像区域影响了CNN的决策。然而，最能反映CNN内部数据处理过程的区域，是否也最符合人类的理解尚不明确。因此，本研究探讨了人类与CNN的图像分类是否基于相同的解释。为评估这种可解释性差异，人类参与者和CNN分别对以下图像片段进行分类：这些片段被认为最具信息量——或由其他人类（通过眼动追踪和手动选择揭示）确定，或由两种XAI方法（Grad-CAM和XRAI）确定。在三个实验中，人类对这些片段进行分类和评分，同时我们也让CNN对其进行分类。结果表明，无论对人类还是CNN而言，两种XAI方法各自的可解释性均高度依赖于图像类型。此外，人类分类性能在使用人类生成的片段时达到最高，无论这些片段如何生成（即通过眼动或手动选择），而人类片段的类型对CNN分类性能具有显著影响。我们的研究结果警示，不应泛泛而谈解释的可解释性，因为这种可解释性随解释方法、待解释内容以及执行解释的智能体而变化。