In this paper, we argue for a paradigm shift from the current model of explainable artificial intelligence (XAI), which may be counter-productive to better human decision making. In early decision support systems, we assumed that we could give people recommendations and that they would consider them, and then follow them when required. However, research found that people often ignore recommendations because they do not trust them; or perhaps even worse, people follow them blindly, even when the recommendations are wrong. Explainable artificial intelligence mitigates this by helping people to understand how and why models give certain recommendations. However, recent research shows that people do not always engage with explainability tools enough to help improve decision making. The assumption that people will engage with recommendations and explanations has proven to be unfounded. We argue this is because we have failed to account for two things. First, recommendations (and their explanations) take control from human decision makers, limiting their agency. Second, giving recommendations and explanations does not align with the cognitive processes employed by people making decisions. This position paper proposes a new conceptual framework called Evaluative AI for explainable decision support. This is a machine-in-the-loop paradigm in which decision support tools provide evidence for and against decisions made by people, rather than provide recommendations to accept or reject. We argue that this mitigates issues of over- and under-reliance on decision support tools, and better leverages human expertise in decision making.

翻译：本文主张从当前可解释人工智能（XAI）模式进行范式转变，因其可能对人类决策优化产生反效果。在早期的决策支持系统中，我们假设向人们提供建议后，他们能加以考量并在必要时遵循。然而研究发现，人们常因缺乏信任而忽视建议；更糟糕的是，即便建议存在错误，人们仍可能盲目遵从。可解释人工智能通过帮助人们理解模型给出特定建议的机理与缘由，试图缓解此问题。但最新研究表明，用户未必会充分使用可解释性工具来优化决策。我们原本假设人们会主动采纳建议与解释，这一假设已被证明缺乏依据。我们认为，根本原因在于忽视了以下两点：其一，建议（及其解释）削弱了人类决策者的自主权，限制了其能动性；其二，给出建议与解释的方式与人类决策的认知过程相悖。本立场论文提出名为"评估型人工智能"（Evaluative AI）的新型概念框架，用于可解释决策支持。这是一种"人在回环"（human-in-the-loop）范式：决策支持工具向人类决策者提供支持或反对其判断的证据，而非直接给出接受或拒绝的建议。我们论证该范式能有效缓解对决策支持工具的过度依赖与依赖不足问题，并更充分地发挥人类在决策中的专业优势。