Predictive algorithms inform consequential decisions in settings where the outcome is selectively observed given choices made by human decision makers. We propose a unified framework for the robust design and evaluation of predictive algorithms in selectively observed data. We impose general assumptions on how much the outcome may vary on average between unselected and selected units conditional on observed covariates and identified nuisance parameters, formalizing popular empirical strategies for imputing missing data such as proxy outcomes and instrumental variables. We develop debiased machine learning estimators for the bounds on a large class of predictive performance estimands, such as the conditional likelihood of the outcome, a predictive algorithm's mean square error, true/false positive rate, and many others, under these assumptions. In an administrative dataset from a large Australian financial institution, we illustrate how varying assumptions on unobserved confounding leads to meaningful changes in default risk predictions and evaluations of credit scores across sensitive groups.

翻译：预测算法在决策结果因人类决策者的选择而被选择性观测的情境中，为关键决策提供依据。我们提出一个统一框架，用于在选择性观测数据中稳健设计与评估预测算法。我们施加一般性假设，限定未选择单元与选择单元之间结果在给定观测协变量与已识别 nuisance 参数条件下的平均变异程度，从而形式化了代理结果与工具变量等常见缺失数据插补经验策略。在此假设下，我们为包括结果条件似然、预测算法均方误差、真/假阳性率等在内的广泛预测性能估计量的界，开发了去偏机器学习估计器。利用澳大利亚某大型金融机构的行政数据集，我们展示了未观测混杂假设的变化如何导致违约风险预测及信用评分在敏感群体间评估的显著变化。