In this paper, we argue for a paradigm shift from the current model of explainable artificial intelligence (XAI), which may be counter-productive to better human decision making. In early decision support systems, we assumed that we could give people recommendations and that they would consider them, and then follow them when required. However, research found that people often ignore recommendations because they do not trust them; or perhaps even worse, people follow them blindly, even when the recommendations are wrong. Explainable artificial intelligence mitigates this by helping people to understand how and why models give certain recommendations. However, recent research shows that people do not always engage with explainability tools enough to help improve decision making. The assumption that people will engage with recommendations and explanations has proven to be unfounded. We argue this is because we have failed to account for two things. First, recommendations (and their explanations) take control from human decision makers, limiting their agency. Second, giving recommendations and explanations does not align with the cognitive processes employed by people making decisions. This position paper proposes a new conceptual framework called Evaluative AI for explainable decision support. This is a machine-in-the-loop paradigm in which decision support tools provide evidence for and against decisions made by people, rather than provide recommendations to accept or reject. We argue that this mitigates issues of over- and under-reliance on decision support tools, and better leverages human expertise in decision making.

翻译：本文主张对当前可解释人工智能（XAI）模型进行范式转换，因为该模型可能对人类决策质量的提升产生反效果。在早期决策支持系统中，我们假设向用户提供建议后，用户会考虑这些建议并在必要时采纳。然而研究表明，人们常因缺乏信任而忽视建议；更糟的是，即便建议错误，人们也会盲目遵从。可解释人工智能通过帮助人们理解模型为何及如何给出特定建议来缓解这一问题。但最新研究表明，用户未必会投入足够精力使用可解释性工具来优化决策。事实证明，用户会主动参与建议与解释互动的假设缺乏依据。我们认为根本原因在于未能充分考虑两点：第一，建议（及其解释）会削弱人类决策者的自主权；第二，建议与解释的呈现方式与人类决策的认知过程不兼容。本立场论文提出了名为"评估型AI"（Evaluative AI）的新型概念框架，用于可解释决策支持。这是一种"人在回路"范式，其中决策支持工具仅为人类决策提供正反两方面的证据，而非给出接受或拒绝的明确建议。我们论证该框架能有效缓解对决策支持工具的过度依赖与依赖不足问题，并能更充分地发挥人类专家在决策中的专业能力。