XAI refers to the techniques and methods for building AI applications which assist end users to interpret output and predictions of AI models. Black box AI applications in high-stakes decision-making situations, such as medical domain have increased the demand for transparency and explainability since wrong predictions may have severe consequences. Model explainability and interpretability are vital successful deployment of AI models in healthcare practices. AI applications' underlying reasoning needs to be transparent to clinicians in order to gain their trust. This paper presents a systematic review of XAI aspects and challenges in the healthcare domain. The primary goals of this study are to review various XAI methods, their challenges, and related machine learning models in healthcare. The methods are discussed under six categories: Features-oriented methods, global methods, concept models, surrogate models, local pixel-based methods, and human-centric methods. Most importantly, the paper explores XAI role in healthcare problems to clarify its necessity in safety-critical applications. The paper intends to establish a comprehensive understanding of XAI-related applications in the healthcare field by reviewing the related experimental results. To facilitate future research for filling research gaps, the importance of XAI models from different viewpoints and their limitations are investigated.

翻译：可解释人工智能（XAI）指构建人工智能应用的技术与方法，旨在帮助最终用户理解AI模型的输出与预测。在医疗等高风险决策场景中，黑箱AI应用的错误预测可能引发严重后果，因此对透明度和可解释性的需求日益增加。模型的可解释性与可理解性是AI模型成功应用于医疗实践的关键。AI应用的底层推理逻辑需对临床医生透明，以建立其信任。本文对医疗领域的XAI问题与挑战进行了系统性综述。研究主要目标为梳理医疗领域的各类XAI方法、挑战及相关机器学习模型。方法分为六类进行讨论：特征导向方法、全局方法、概念模型、代理模型、局部像素方法及人本方法。尤其重要的是，本文探讨了XAI在医疗问题中的角色，阐明其在安全关键型应用中的必要性。通过回顾相关实验结果，本文旨在建立对医疗领域XAI应用的全面理解。为填补研究空白以促进未来研究，本文还从不同视角分析了XAI模型的重要性及其局限性。