We consider the problem of evaluating the performance of a decision policy using past observational data. The outcome of a policy is measured in terms of a loss (aka. disutility or negative reward) and the main problem is making valid inferences about its out-of-sample loss when the past data was observed under a different and possibly unknown policy. Using a sample-splitting method, we show that it is possible to draw such inferences with finite-sample coverage guarantees about the entire loss distribution, rather than just its mean. Importantly, the method takes into account model misspecifications of the past policy - including unmeasured confounding. The evaluation method can be used to certify the performance of a policy using observational data under a specified range of credible model assumptions.

翻译：我们考虑利用历史观测数据评估决策策略性能的问题。策略的效果通过损失（亦称负效用或负奖励）来衡量，核心问题是在历史数据由不同且可能未知的策略收集时，对其样本外损失进行有效推断。通过采用样本分割方法，我们证明可以针对整个损失分布（而非仅其均值）进行具有有限样本覆盖保证的推断。重要的是，该方法考虑了历史策略的模型误设（包括未测量的混杂因素）。该评估方法可在指定可信模型假设范围内，利用观测数据对策略性能进行认证。