Explainable Artificial Intelligence (XAI) is essential for the transparency and clinical adoption of Clinical Decision Support Systems (CDSS). However, the real-world effectiveness of existing XAI methods remains limited and is inconsistently evaluated. This study conducts a systematic PRISMA-guided survey of 31 human-centered evaluations (HCE) of XAI applied to CDSS, classifying them by XAI methodology, evaluation design, and adoption barrier. Our findings reveal that most existing studies employ post-hoc, model-agnostic approaches such as SHAP and Grad-CAM, typically assessed through small-scale clinician studies. The results show that over 80% of the studies adopt post-hoc, model-agnostic approaches such as SHAP and Grad-CAM, and that clinician sample sizes remain below 25 participants. The findings indicate that explanations generally improve clinician trust and diagnostic confidence, but frequently increase cognitive load and exhibit misalignment with domain reasoning processes. To bridge these gaps, we propose a stakeholder-centric evaluation framework that integrates socio-technical principles and human-computer interaction to guide the future development of clinically viable and trustworthy XAI-based CDSS.

翻译：可解释人工智能（XAI）对于临床决策支持系统（CDSS）的透明度和临床采用至关重要。然而，现有XAI方法在真实场景中的有效性仍然有限，且评估方式缺乏一致性。本研究通过PRISMA指南对31项应用于CDSS的XAI人本评估（HCE）进行了系统性综述，依据XAI方法学、评估设计和应用障碍进行分类。研究发现，现有研究大多采用SHAP和Grad-CAM等事后模型无关方法，通常通过小规模临床医师研究进行评估。结果显示超过80%的研究采用SHAP和Grad-CAM等事后模型无关方法，且临床医师样本量普遍低于25人。研究结果表明，解释机制总体上能提升临床医师的信任度和诊断信心，但常会增加认知负荷，并与领域推理过程存在偏差。为弥合这些差距，我们提出了一个以利益相关者为中心的评估框架，该框架整合了社会技术原则和人机交互理念，以指导未来开发具有临床可行性和可信度的基于XAI的CDSS。