Developing accurate off-policy estimators is crucial for both evaluating and optimizing for new policies. The main challenge in off-policy estimation is the distribution shift between the logging policy that generates data and the target policy that we aim to evaluate. Typically, techniques for correcting distribution shift involve some form of importance sampling. This approach results in unbiased value estimation but often comes with the trade-off of high variance, even in the simpler case of one-step contextual bandits. Furthermore, importance sampling relies on the common support assumption, which becomes impractical when the action space is large. To address these challenges, we introduce the Policy Convolution (PC) family of estimators. These methods leverage latent structure within actions -- made available through action embeddings -- to strategically convolve the logging and target policies. This convolution introduces a unique bias-variance trade-off, which can be controlled by adjusting the amount of convolution. Our experiments on synthetic and benchmark datasets demonstrate remarkable mean squared error (MSE) improvements when using PC, especially when either the action space or policy mismatch becomes large, with gains of up to 5 - 6 orders of magnitude over existing estimators.

翻译：开发精确的离线策略评估器对于新策略的评估与优化至关重要。离线策略评估的主要挑战在于生成数据的记录策略与待评估目标策略之间存在分布偏移。通常，校正分布偏移的技术涉及某种形式的重要性采样。这种方法能实现无偏的价值估计，但往往伴随着高方差的权衡，即使在单步上下文赌博机这一简化情形下也是如此。此外，重要性采样依赖于共同支撑假设，当动作空间较大时该假设变得不切实际。为应对这些挑战，我们引入了策略卷积（PC）估计器家族。这些方法利用动作嵌入所揭示的动作潜在结构，对记录策略与目标策略进行策略性卷积。这种卷积引入了独特的偏差-方差权衡，可通过调整卷积量加以控制。在合成数据集与基准数据集上的实验表明，使用PC时均方误差（MSE）得到了显著改善，尤其是在动作空间或策略失配程度较大时，相比现有估计器性能提升可达5至6个数量级。