A growing body of research has explored how to support humans in making better use of AI-based decision support, including via training and onboarding. Existing research has focused on decision-making tasks where it is possible to evaluate "appropriate reliance" by comparing each decision against a ground truth label that cleanly maps to both the AI's predictive target and the human decision-maker's goals. However, this assumption does not hold in many real-world settings where AI tools are deployed today (e.g., social work, criminal justice, and healthcare). In this paper, we introduce a process-oriented notion of appropriate reliance called critical use that centers the human's ability to situate AI predictions against knowledge that is uniquely available to them but unavailable to the AI model. To explore how training can support critical use, we conduct a randomized online experiment in a complex social decision-making setting: child maltreatment screening. We find that, by providing participants with accelerated, low-stakes opportunities to practice AI-assisted decision-making in this setting, novices came to exhibit patterns of disagreement with AI that resemble those of experienced workers. A qualitative examination of participants' explanations for their AI-assisted decisions revealed that they drew upon qualitative case narratives, to which the AI model did not have access, to learn when (not) to rely on AI predictions. Our findings open new questions for the study and design of training for real-world AI-assisted decision-making.

翻译：越来越多的研究探讨如何通过培训与入职引导等方式，支持人类更有效地利用基于AI的决策支持系统。现有研究主要聚焦于可评估“适当依赖”的决策任务——通过将每个决策与明确映射到AI预测目标和人类决策者目标的真实标签进行对比。然而，这一假设在当今AI工具实际部署的诸多场景（如社会工作、刑事司法和医疗保健）中并不成立。本文提出了一种以过程为导向的“适当依赖”概念——“批判性使用”，其核心在于人类能够将AI预测置于AI模型无法获取、但人类自身独有的知识体系中进行定位。为探究培训如何支持批判性使用，我们在儿童虐待筛查这一复杂社会决策场景中开展了一项随机在线实验。研究发现，通过为新手提供该场景下低风险、高密度的AI辅助决策实践机会，他们逐渐展现出与经验丰富工作者相似的与AI预测结果的偏离模式。对参与者AI辅助决策解释的定性分析表明，他们借助AI模型无法获取的定性案例叙述，学习何时（不）应依赖AI预测。我们的发现为真实世界AI辅助决策培训的研究与设计开辟了新的问题方向。