Predictive models are often introduced to decision-making tasks under the rationale that they improve performance over an existing decision-making policy. However, it is challenging to compare predictive performance against an existing decision-making policy that is generally under-specified and dependent on unobservable factors. These sources of uncertainty are often addressed in practice by making strong assumptions about the data-generating mechanism. In this work, we propose a method to compare the predictive performance of decision policies under a variety of modern identification approaches from the causal inference and off-policy evaluation literatures (e.g., instrumental variable, marginal sensitivity model, proximal variable). Key to our method is the insight that there are regions of uncertainty that we can safely ignore in the policy comparison. We develop a practical approach for finite-sample estimation of regret intervals under no assumptions on the parametric form of the status quo policy. We verify our framework theoretically and via synthetic data experiments. We conclude with a real-world application using our framework to support a pre-deployment evaluation of a proposed modification to a healthcare enrollment policy.

翻译：预测模型常被引入决策任务中，其理由是其能够比现有决策策略带来性能提升。然而，与通常涉及未完全指定且依赖于不可观测因素的现有决策策略进行预测性能比较颇具挑战性。实践中，这些不确定性来源常通过对数据生成机制做出强假设来处理。本文提出一种方法，用于在因果推断和离线策略评估文献中的多种现代识别方法（如工具变量、边际敏感度模型、近端变量）下比较决策策略的预测性能。该方法的关键洞见在于：策略比较中存在可安全忽略的不确定性区域。我们开发了一种实用方法，无需对现状策略的参数形式做任何假设，即可实现有限样本下的遗憾区间估计。通过理论分析和合成数据实验验证了该框架，并最终将其应用于一项真实世界的医疗注册策略拟议变更的部署前评估。