A growing literature on human-AI decision-making investigates strategies for combining human judgment with statistical models to improve decision-making. Research in this area often evaluates proposed improvements to models, interfaces, or workflows by demonstrating improved predictive performance on "ground truth" labels. However, this practice overlooks a key difference between human judgments and model predictions. Whereas humans reason about broader phenomena of interest in a decision -- including latent constructs that are not directly observable, such as disease status, the "toxicity" of online comments, or future "job performance" -- predictive models target proxy labels that are readily available in existing datasets. Predictive models' reliance on simplistic proxies makes them vulnerable to various sources of statistical bias. In this paper, we identify five sources of target variable bias that can impact the validity of proxy labels in human-AI decision-making tasks. We develop a causal framework to disentangle the relationship between each bias and clarify which are of concern in specific human-AI decision-making tasks. We demonstrate how our framework can be used to articulate implicit assumptions made in prior modeling work, and we recommend evaluation strategies for verifying whether these assumptions hold in practice. We then leverage our framework to re-examine the designs of prior human subjects experiments that investigate human-AI decision-making, finding that only a small fraction of studies examine factors related to target variable bias. We conclude by discussing opportunities to better address target variable bias in future research.

翻译：关于人机协同决策的研究日益增多，这些研究探索如何将人类判断与统计模型相结合以提升决策质量。该领域的研究常通过展示模型在"真实标签"上预测性能的提升，来评估对模型、界面或工作流程的改进。然而，这种做法忽视了人类判断与模型预测之间的关键差异：人类会思考决策中更广泛的潜在现象——包括不可直接观测的隐性构念，如疾病状态、在线评论的"毒性"或未来"工作绩效"——而预测模型针对的是现有数据集中易获取的代理标签。预测模型对简化代理标签的依赖使其容易受到各类统计偏差的影响。本文识别了五种可能影响人机协同决策任务中代理标签有效性的目标变量偏差，并构建因果框架以厘清各类偏差之间的关系，阐明在具体人机协同任务中应重点关注哪些偏差。我们展示了该框架如何用于阐明以往建模研究中的隐含假设，并提出验证这些假设在实践中是否成立的评估策略。随后，我们运用该框架重新审视以往探究人机协同决策的人类受试者实验设计，发现仅少数研究考察了与目标变量偏差相关的因素。最后，我们讨论了未来研究更好地应对目标变量偏差的机遇。