Understanding how a prediction model will perform in a new environment before deployment is essential to preventing harm when algorithms inform decision-making. Two common sources of model performance degradation are (i) covariate shift, where the target covariate distribution differs from the source, and (ii) selective labels, where the observability of outcomes depends on historical decisions. We study pre-deployment model evaluation under the joint presence of covariate shift and labeling of outcomes selectively based on observed features. In particular, we present a double machine learning procedure for estimating the target risk of an arbitrary black-box prediction model under a general loss function. We show identification of this estimand under standard assumptions and derive a bias-corrected estimator based on the influence function of the target risk. Finally, we evaluate our estimator through experiments using the eICU electronic health records database, showing that it tracks the true target risk more accurately than methods that address either selective labels or covariate shift alone, as well as baselines that combine standard plug-in approaches.

翻译：理解预测模型在部署前于新环境中的表现，对于防止算法辅助决策时造成危害至关重要。导致模型性能下降的两个常见原因是：（i）协变量偏移，即目标协变量分布与源分布不同；（ii）选择性标签，即结果的可观测性取决于历史决策。我们研究在协变量偏移和基于观测特征选择性标记结果共同存在下的部署前模型评估问题。具体而言，我们提出了一种双重机器学习程序，用于在一般损失函数下估计任意黑箱预测模型的目标风险。我们在标准假设下证明了该估计量的可识别性，并基于目标风险的影响函数推导出偏差校正估计量。最后，我们利用eICU电子健康记录数据库通过实验评估该估计量，结果表明，相较于仅处理选择性标签或协变量偏移的方法，以及结合标准插件方法的基线，该估计量能更准确地追踪真实目标风险。