Reverse Kullback-Leibler (KL) divergence-based regularization with respect to a fixed reference policy is widely used in modern reinforcement learning to preserve the desired traits of the reference policy and sometimes to promote exploration (using uniform reference policy, known as entropy regularization). Beyond serving as a mere anchor, the reference policy can also be interpreted as encoding prior knowledge about good actions in the environment. In the context of alignment, recent game-theoretic approaches have leveraged KL regularization with pretrained language models as reference policies, achieving notable empirical success in self-play methods. Despite these advances, the theoretical benefits of KL regularization in game-theoretic settings remain poorly understood. In this work, we develop and analyze algorithms that provably achieve improved sample efficiency under KL regularization. We study both two-player zero-sum matrix games and Markov games: for matrix games, we propose OMG, an algorithm based on best response sampling with optimistic bonuses, and extend this idea to Markov games through the algorithm SOMG, which also uses best response sampling and a novel concept of superoptimistic bonuses. Both algorithms achieve a logarithmic regret in $T$ that scales inversely with the KL regularization strength $β$ in addition to the traditional $\widetilde{\mathcal{O}}(\sqrt{T})$ regret without the $β^{-1}$ dependence.

翻译：基于固定参考策略的反向Kullback-Leibler（KL）散度正则化在现代强化学习中被广泛使用，以保持参考策略的期望特性，有时也用于促进探索（使用均匀参考策略，即熵正则化）。参考策略不仅充当锚点，也可被解释为编码了关于环境中优良动作的先验知识。在对齐的背景下，最近的博弈论方法已利用以预训练语言模型作为参考策略的KL正则化，在自博弈方法中取得了显著的实证成功。尽管取得了这些进展，KL正则化在博弈论环境中的理论优势仍未得到充分理解。在本工作中，我们开发并分析了在KL正则化下可证明实现更高样本效率的算法。我们同时研究了双人零和矩阵博弈与马尔可夫博弈：对于矩阵博弈，我们提出了OMG算法，该算法基于带有乐观奖励的最佳响应采样；并将这一思想通过SOMG算法扩展至马尔可夫博弈，该算法同样使用最佳响应采样及一种新颖的超乐观奖励概念。两种算法均实现了关于$T$的对数遗憾，其缩放比例与KL正则化强度$β$成反比，此外还具备传统的$\widetilde{\mathcal{O}}(\sqrt{T})$遗憾（不含$β^{-1}$依赖项）。