We consider the problem of evaluating the performance of a decision policy using past observational data. The outcome of a policy is measured in terms of a loss (aka. disutility or negative reward) and the main problem is making valid inferences about its out-of-sample loss when the past data was observed under a different and possibly unknown policy. Using a sample-splitting method, we show that it is possible to draw such inferences with finite-sample coverage guarantees about the entire loss distribution, rather than just its mean. Importantly, the method takes into account model misspecifications of the past policy -- including unmeasured confounding. The evaluation method can be used to certify the performance of a policy using observational data under a specified range of credible model assumptions.

翻译：我们考虑利用历史观测数据评估决策策略性能的问题。策略的结果以损失（即非效用或负奖励）衡量，核心问题是在历史数据由不同且可能未知的策略观测所得时，对其样本外损失进行有效推断。通过样本拆分方法，我们证明能在有限样本下对整个损失分布（而非仅其均值）提供覆盖保证的推断。重要的是，该方法考虑了历史策略的模型误设定（包括未测量的混淆因素）。该评估方法可在指定可信模型假设范围内，利用观测数据对策略性能进行认证。