Explainable AI (XAI) is widely viewed as a sine qua non for ever-expanding AI research. A better understanding of the needs of XAI users, as well as human-centered evaluations of explainable models are both a necessity and a challenge. In this paper, we explore how HCI and AI researchers conduct user studies in XAI applications based on a systematic literature review. After identifying and thoroughly analyzing 97core papers with human-based XAI evaluations over the past five years, we categorize them along the measured characteristics of explanatory methods, namely trust, understanding, usability, and human-AI collaboration performance. Our research shows that XAI is spreading more rapidly in certain application domains, such as recommender systems than in others, but that user evaluations are still rather sparse and incorporate hardly any insights from cognitive or social sciences. Based on a comprehensive discussion of best practices, i.e., common models, design choices, and measures in user studies, we propose practical guidelines on designing and conducting user studies for XAI researchers and practitioners. Lastly, this survey also highlights several open research directions, particularly linking psychological science and human-centered XAI.

翻译：可解释人工智能（XAI）被广泛视为不断扩展的AI研究的必要条件。更好地理解XAI用户的需求，以及对可解释模型进行以人为中心的评估，既是必然要求也是重大挑战。本文基于系统性文献综述，探讨了人机交互（HCI）和AI研究人员如何在XAI应用中开展用户研究。通过识别并深入分析过去五年间97篇基于人类评估的核心XAI论文，我们根据解释方法所衡量的特征维度（即信任度、理解度、可用性及人机协作性能）对其进行了分类。研究表明，XAI在推荐系统等特定应用领域的普及速度显著快于其他领域，但用户评估仍较为稀缺，且几乎未融入认知科学或社会科学的洞见。基于对用户研究中最佳实践（即通用模型、设计选择与评估指标）的全面讨论，我们为XAI研究人员和实践者提出了关于设计与实施用户研究的实用指南。最后，本综述还指出了若干开放研究方向，特别是心理学科学与以人为本的XAI之间的交叉融合。