In this paper, we argue for a paradigm shift from the current model of explainable artificial intelligence (XAI), which may be counter-productive to better human decision making. In early decision support systems, we assumed that we could give people recommendations and that they would consider them, and then follow them when required. However, research found that people often ignore recommendations because they do not trust them; or perhaps even worse, people follow them blindly, even when the recommendations are wrong. Explainable artificial intelligence mitigates this by helping people to understand how and why models give certain recommendations. However, recent research shows that people do not always engage with explainability tools enough to help improve decision making. The assumption that people will engage with recommendations and explanations has proven to be unfounded. We argue this is because we have failed to account for two things. First, recommendations (and their explanations) take control from human decision makers, limiting their agency. Second, giving recommendations and explanations does not align with the cognitive processes employed by people making decisions. This position paper proposes a new conceptual framework called Evaluative AI for explainable decision support. This is a machine-in-the-loop paradigm in which decision support tools provide evidence for and against decisions made by people, rather than provide recommendations to accept or reject. We argue that this mitigates issues of over- and under-reliance on decision support tools, and better leverages human expertise in decision making.

翻译：本文主张对当前可解释人工智能（XAI）范式进行根本性转变，因为现有模式可能反而不利于人类决策质量的提升。在早期决策支持系统中，我们假设通过向人们提供建议，他们会对此进行考量并在必要时采纳。但研究发现，人们常因缺乏信任而忽略建议，甚至更糟的是——当建议本身错误时仍盲目跟从。可解释人工智能通过帮助用户理解模型给出特定建议的内在机理与逻辑来缓解上述问题。然而最新研究表明，用户未必会充分使用可解释性工具来优化决策过程。我们假设用户会主动交互建议与解释的预设已被证伪。我们认为这是源于两个关键因素未被纳入考量：其一，建议（及其解释）剥夺了人类决策者的自主权，限制了其能动性；其二，提供建议与解释的机制不符合人类决策的认知模式。本立场论文提出名为“评估型人工智能”（Evaluative AI）的新型概念框架，这是一种“人在回路”的决策支持范式，其中决策支持工具仅呈现支持或反对人类决策的证据，而非直接给出接受或拒绝的推荐建议。我们认为该方案可缓解对决策支持工具的过度依赖与低度依赖问题，并更高效地整合人类决策的专业智慧。