Machine learning is increasingly used to select which individuals receive limited-resource interventions in domains such as human services, education, development, and more. However, it is often not apparent what the right quantity is for models to predict. In particular, policymakers rarely have access to data from a randomized controlled trial (RCT) that would enable accurate estimates of treatment effects -- which individuals would benefit more from the intervention. Observational data is more likely to be available, creating a substantial risk of bias in treatment effect estimates. Practitioners instead commonly use a technique termed "risk-based targeting" where the model is just used to predict each individual's status quo outcome (an easier, non-causal task). Those with higher predicted risk are offered treatment. There is currently almost no empirical evidence to inform which choices lead to the most effect machine learning-informed targeting strategies in social domains. In this work, we use data from 5 real-world RCTs in a variety of domains to empirically assess such choices. We find that risk-based targeting is almost always inferior to targeting based on even biased estimates of treatment effects. Moreover, these results hold even when the policymaker has strong normative preferences for assisting higher-risk individuals. Our results imply that, despite the widespread use of risk prediction models in applied settings, practitioners may be better off incorporating even weak evidence about heterogeneous causal effects to inform targeting.

翻译：机器学习越来越多地应用于人类服务、教育、发展等领域，以选择哪些个体获得有限资源的干预措施。然而，模型应预测何种恰当的量值往往并不明确。特别是，政策制定者很少能获得来自随机对照试验（RCT）的数据，而此类数据能够准确估计处理效应——即哪些个体能从干预中获益更多。观察性数据更可能被获取，但这会导致处理效应估计存在显著的偏倚风险。实践中，从业者通常采用一种称为“基于风险的目标定位”技术，即模型仅用于预测每个个体的现状结果（一项更简单、非因果性的任务）。那些预测风险较高的个体被提供干预。目前，几乎没有实证证据能够说明在社会领域，哪些选择能产生最有效的机器学习辅助目标定位策略。在本研究中，我们利用来自五个不同领域真实世界随机对照试验的数据，对这些选择进行实证评估。我们发现，基于风险的目标定位几乎总是劣于基于（即使是存在偏倚的）处理效应估计的目标定位。此外，即使政策制定者对援助高风险个体有强烈的规范性偏好，这些结论依然成立。我们的结果表明，尽管风险预测模型在应用环境中被广泛使用，但从业者或许应整合关于异质性因果效应的证据（即使是较弱的证据）来指导目标定位，这可能会带来更好的效果。