A growing literature on human-AI decision-making investigates strategies for combining human judgment with statistical models to improve decision-making. Research in this area often evaluates proposed improvements to models, interfaces, or workflows by demonstrating improved predictive performance on "ground truth" labels. However, this practice overlooks a key difference between human judgments and model predictions. Whereas humans reason about broader phenomena of interest in a decision - including latent constructs that are not directly observable, such as disease status, the "toxicity" of online comments, or future "job performance" - predictive models target proxy labels that are readily available in existing datasets. Predictive models' reliance on simplistic proxies makes them vulnerable to various sources of statistical bias. In this paper, we identify five sources of target variable bias that can impact the validity of proxy labels in human-AI decision-making tasks. We develop a causal framework to disentangle the relationship between each bias and clarify which are of concern in specific human-AI decision-making tasks. We demonstrate how our framework can be used to articulate implicit assumptions made in prior modeling work, and we recommend evaluation strategies for verifying whether these assumptions hold in practice. We then leverage our framework to re-examine the designs of prior human subjects experiments that investigate human-AI decision-making, finding that only a small fraction of studies examine factors related to target variable bias. We conclude by discussing opportunities to better address target variable bias in future research.

翻译：关于人机协同决策的文献日益增多，探讨如何将人类判断与统计模型结合以提升决策效能。该领域研究常通过证明模型在"真值"标签上预测性能的提升来评估对模型、界面或工作流的改进。然而，这种做法忽略了人类判断与模型预测之间的关键差异：人类推理关注决策中更广泛的现象——包括无法直接观测的潜在构念（如疾病状态、网络评论的"毒性"、未来"工作绩效"），而预测模型往往针对现有数据集中易于获取的代理标签。预测模型对简化代理标签的依赖使其易受多种统计偏差影响。本文识别出可能影响人机决策任务中代理标签有效性的五类目标变量偏差，并构建因果框架厘清各类偏差间的关系，阐明具体人机决策任务中需关注哪些偏差。我们展示了该框架如何揭示既往建模工作中隐含假设，并推荐用于验证这些假设在实践中是否成立的评估策略。继而运用该框架重新审视既往人机决策实验设计，发现仅少数研究考察了与目标变量偏差相关的因素。最后，我们讨论了未来研究更好应对目标变量偏差的可行方向。