With an increase in deep learning-based methods, the call for explainability of such methods grows, especially in high-stakes decision making areas such as medical image analysis. This survey presents an overview of eXplainable Artificial Intelligence (XAI) used in deep learning-based medical image analysis. A framework of XAI criteria is introduced to classify deep learning-based medical image analysis methods. Papers on XAI techniques in medical image analysis are then surveyed and categorized according to the framework and according to anatomical location. The paper concludes with an outlook of future opportunities for XAI in medical image analysis.

翻译：随着以深层次学习为基础的方法的增加,人们日益要求解释这些方法,特别是在医学图像分析等高层次决策领域。本调查概述了在深层次学习基础上医学图像分析中使用的可复制人工智能(XAI),采用了XAI标准框架,对基于深层次学习的医疗图像分析方法进行分类,然后根据框架和解剖学地点对医学图像分析中XAI技术的论文进行调查和分类。论文最后展望了XAI在医学图像分析中的未来机会。