A growing literature on human-AI decision-making investigates strategies for combining human judgment with statistical models to improve decision-making. Research in this area often evaluates proposed improvements to models, interfaces, or workflows by demonstrating improved predictive performance on "ground truth" labels. However, this practice overlooks a key difference between human judgments and model predictions. Whereas humans reason about broader phenomena of interest in a decision -- including latent constructs that are not directly observable, such as disease status, the "toxicity" of online comments, or future "job performance" -- predictive models target proxy labels that are readily available in existing datasets. Predictive models' reliance on simplistic proxies makes them vulnerable to various sources of statistical bias. In this paper, we identify five sources of target variable bias that can impact the validity of proxy labels in human-AI decision-making tasks. We develop a causal framework to disentangle the relationship between each bias and clarify which are of concern in specific human-AI decision-making tasks. We demonstrate how our framework can be used to articulate implicit assumptions made in prior modeling work, and we recommend evaluation strategies for verifying whether these assumptions hold in practice. We then leverage our framework to re-examine the designs of prior human subjects experiments that investigate human-AI decision-making, finding that only a small fraction of studies examine factors related to target variable bias. We conclude by discussing opportunities to better address target variable bias in future research.

翻译：关于人机协同决策的文献日益增多，探讨如何将人类判断与统计模型相结合以提升决策质量。该领域的研究常通过展示模型在“真实标签”上的预测性能改进，来评估对模型、界面或流程提出的优化方案。然而，这种做法忽略了人类判断与模型预测之间的关键差异：人类会推演决策中涉及更广泛的现象——包括无法直接观测的潜变量，如疾病状态、网络评论的“有害性”或未来“工作绩效”——而预测模型则针对现有数据集中易于获取的代理标签。预测模型对简化代理的依赖使其易受多种统计偏差影响。本文识别了五种可能影响人机协同决策任务中代理标签效度的目标变量偏差来源，并构建了一个因果框架以厘清各类偏差间的关联，阐明特定人机协同任务中应重点关注哪些偏差。我们展示了如何利用该框架阐明先前建模研究中隐含的假设，并推荐了验证这些假设在实践中是否成立的评估策略。进而，我们运用该框架重新审视以往探究人机协同决策的人类受试者实验设计，发现仅少数研究涉及与目标变量偏差相关的因素。最后，我们讨论了未来研究中更好应对目标变量偏差的机遇。