Reinforcement Learning with Verified Reward (RLVR) has emerged as a critical paradigm for advancing the reasoning capabilities of Large Language Models (LLMs). Most existing RLVR methods, such as GRPO and its variants, ensure stable updates by constraining policy divergence through clipping likelihood ratios. This paper introduces a unified clipping framework that characterizes existing methods via a general notion of policy divergence, encompassing both likelihood ratios and Kullback-Leibler (KL) divergences and extending to alternative measures. The framework provides a principled foundation for systematically analyzing how different policy divergence measures affect exploration and performance. We further identify the KL3 estimator, a variance-reduced Monte Carlo estimator of the KL divergence, as a key policy divergence constraint. We theoretically demonstrate that the KL3-based constraint is mathematically equivalent to an asymmetric ratio-based clipping that reallocates probability mass toward high-confidence actions, promoting stronger exploration while retaining the simplicity of GRPO-style methods. Empirical results on mathematical reasoning benchmarks demonstrate that incorporating the KL3 estimator into GRPO improves both training stability and final performance, highlighting the importance of principled policy divergence constraints in policy optimization.

翻译：带验证奖励的强化学习已成为提升大型语言模型推理能力的关键范式。现有RLVR方法（如GRPO及其变体）大多通过裁剪似然比来约束策略差异，从而确保稳定更新。本文提出统一的裁剪框架，通过广义策略差异度量来刻画现有方法——该框架不仅涵盖似然比与KL散度，还可扩展至其他度量方式。该框架为系统分析不同策略差异度量如何影响探索与性能提供了理论依据。我们进一步提出KL3估计器（一种方差缩减的KL散度蒙特卡洛估计器）作为关键策略差异约束。理论证明表明：基于KL3的约束在数学上等价于非对称比率裁剪，能将概率质量重新分配至高置信度动作，在保持GRPO类方法简洁性的同时增强探索能力。数学推理基准测试表明，将KL3估计器融入GRPO能同时提升训练稳定性与最终性能，印证了策略优化中理论化策略差异约束的重要性。