Explainable AI (XAI) is widely viewed as a sine qua non for ever-expanding AI research. A better understanding of the needs of XAI users, as well as human-centered evaluations of explainable models are both a necessity and a challenge. In this paper, we explore how HCI and AI researchers conduct user studies in XAI applications based on a systematic literature review. After identifying and thoroughly analyzing 97core papers with human-based XAI evaluations over the past five years, we categorize them along the measured characteristics of explanatory methods, namely trust, understanding, usability, and human-AI collaboration performance. Our research shows that XAI is spreading more rapidly in certain application domains, such as recommender systems than in others, but that user evaluations are still rather sparse and incorporate hardly any insights from cognitive or social sciences. Based on a comprehensive discussion of best practices, i.e., common models, design choices, and measures in user studies, we propose practical guidelines on designing and conducting user studies for XAI researchers and practitioners. Lastly, this survey also highlights several open research directions, particularly linking psychological science and human-centered XAI.

翻译：可解释人工智能（XAI）被广泛视为人工智能研究持续扩展的必要条件。更好地理解XAI用户的需求，以及对可解释模型进行以人为中心的评估，既是一项必要任务，也是一大挑战。本文基于系统性文献综述，探讨了人机交互与人工智能研究者如何在XAI应用中进行用户研究。通过识别并深入分析过去五年间97篇包含基于人类的XAI评估的核心论文，我们依据解释方法所衡量的特性——即信任度、理解度、可用性以及人机协作性能——对这些研究进行了分类。我们的研究表明，XAI在某些应用领域（如推荐系统）的普及速度比其他领域更快，但用户评估仍然相当稀疏，且几乎未纳入认知科学或社会科学的见解。基于对最佳实践（即用户研究中常见的模型、设计选择与测量方法）的全面讨论，我们为XAI研究者与实践者提出了设计和实施用户研究的实用指南。最后，本综述还强调了若干开放的研究方向，特别是心理学科学与以人为中心的XAI之间的关联。