Being able to explain the prediction to clinical end-users is a necessity to leverage the power of artificial intelligence (AI) models for clinical decision support. For medical images, a feature attribution map, or heatmap, is the most common form of explanation that highlights important features for AI models' prediction. However, it is unknown how well heatmaps perform on explaining decisions on multi-modal medical images, where each image modality or channel visualizes distinct clinical information of the same underlying biomedical phenomenon. Understanding such modality-dependent features is essential for clinical users' interpretation of AI decisions. To tackle this clinically important but technically ignored problem, we propose the modality-specific feature importance (MSFI) metric. It encodes clinical image and explanation interpretation patterns of modality prioritization and modality-specific feature localization. We conduct a clinical requirement-grounded, systematic evaluation using computational methods and a clinician user study. Results show that the examined 16 heatmap algorithms failed to fulfill clinical requirements to correctly indicate AI model decision process or decision quality. The evaluation and MSFI metric can guide the design and selection of XAI algorithms to meet clinical requirements on multi-modal explanation.

翻译：能够向临床终端用户解释预测结果是利用人工智能模型支持临床决策的必要条件。在医学影像中，特征归因图（或称热力图）是最常见的解释形式，用于突出对AI模型预测至关重要的特征。然而，尚不清楚热力图在多模态医学影像决策解释中的表现——在多模态影像中，每种影像模态或通道可视化展示了同一生物医学现象的不同临床信息。理解此类模态依赖特征对临床用户解读AI决策至关重要。为解决这一临床关键但技术上被忽视的问题，我们提出了模态特异性特征重要性（MSFI）指标。该指标编码了临床影像与解释解读模式中的模态优先级排序及模态特异性特征定位。我们通过计算方法和临床医师用户研究，开展了基于临床需求的系统性评估。结果表明，所考察的16种热力图算法均未能满足临床需求，无法正确指示AI模型决策过程或决策质量。本评估及MSFI指标可指导可解释人工智能算法的设计与选择，以满足多模态解释的临床需求。