The recent success and openness of DeepSeek-R1 have brought widespread attention to Group Relative Policy Optimization (GRPO) as a reinforcement learning method for large reasoning models (LRMs). In this work, we analyze the GRPO objective under a binary reward setting and reveal an inherent limitation of question-level difficulty bias. We also identify a connection between GRPO and traditional discriminative methods in supervised learning. Motivated by these insights, we introduce a new Discriminative Constrained Optimization (DisCO) framework for reinforcing LRMs, grounded in the principle of discriminative learning. The main differences between DisCO and GRPO and its recent variants are: (1) it replaces the group relative objective with a discriminative objective defined by a scoring function; (2) it abandons clipping-based surrogates in favor of non-clipping RL surrogate objectives used as scoring functions; (3) it employs a simple yet effective constrained optimization approach to enforce the KL divergence constraint. As a result, DisCO offers notable advantages over GRPO and its variants: (i) it completely eliminates difficulty bias by adopting discriminative objectives; (ii) it addresses the entropy instability in GRPO and its variants through the use of non-clipping scoring functions and a constrained optimization approach, yielding long and stable training dynamics; (iii) it allows the incorporation of advanced discriminative learning techniques to address data imbalance, where a significant number of questions have more negative than positive generated answers during training. Our experiments on enhancing the mathematical reasoning capabilities of SFT-finetuned models show that DisCO significantly outperforms GRPO and its improved variants such as DAPO, achieving average gains of 7\% over GRPO and 6\% over DAPO across six benchmark tasks for a 1.5B model.

翻译：DeepSeek-R1 的成功与开源使群体相对策略优化（GRPO）作为一种面向大型推理模型（LRMs）的强化学习方法受到广泛关注。本文在二元奖励设定下分析了 GRPO 目标函数，揭示了其固有的问题级难度偏差局限，并发现了 GRPO 与监督学习中传统判别式方法之间的关联。基于这些洞察，我们提出了一种基于判别学习原理的新型判别式约束优化（DisCO）框架，用于强化 LRMs。DisCO 与 GRPO 及其近期变体的主要区别在于：（1）用评分函数定义的判别式目标替代群体相对目标；（2）放弃基于裁剪的代理目标，转而采用作为评分函数的非裁剪强化学习代理目标；（3）采用简洁有效的约束优化方法强制执行 KL 散度约束。因此，DisCO 相比 GRPO 及其变体具有显著优势：（i）通过采用判别式目标完全消除了难度偏差；（ii）借助非裁剪评分函数与约束优化方法解决了 GRPO 及其变体中的熵不稳定性，实现了长周期稳定的训练动态；（iii）允许引入先进的判别学习技术以应对数据不平衡问题，即训练过程中大量问题产生的负面答案多于正面答案的情况。我们在提升 SFT 微调模型数学推理能力的实验表明，DisCO 显著优于 GRPO 及其改进变体（如 DAPO），在 1.5B 参数的模型上，于六项基准任务中平均分别取得 7% 和 6% 的性能提升。