The evolution of Large Language Models (LLMs) has catalyzed a paradigm shift from superficial instruction following to rigorous long-horizon reasoning. While Group Relative Policy Optimization (GRPO) has emerged as a pivotal mechanism for eliciting such post-training reasoning capabilities due to its exceptional performance, it remains plagued by significant training instability and poor sample efficiency. We theoretically identify the root cause of these issues as the lack of distinctiveness within on-policy rollouts: for routine queries, highly homogeneous samples induce destructive gradient conflicts; whereas for hard queries, the scarcity of valid positive samples results in ineffective optimization. To bridge this gap, we propose Distinctiveness-aware Group Relative Policy Optimization (DaGRPO). DaGRPO incorporates two core mechanisms: (1) Sequence-level Gradient Rectification, which utilizes fine-grained scoring to dynamically mask sample pairs with low distinctiveness, thereby eradicating gradient conflicts at the source; and (2) Off-policy Data Augmentation, which introduces high-quality anchors to recover training signals for challenging tasks. Extensive experiments across 9 mathematical reasoning and out-of-distribution (OOD) generalization benchmarks demonstrate that DaGRPO significantly surpasses existing SFT, GRPO, and hybrid baselines, achieving new state-of-the-art performance (e.g., a +4.7% average accuracy gain on math benchmarks). Furthermore, in-depth analysis confirms that DaGRPO effectively mitigates gradient explosion and accelerates the emergence of long-chain reasoning capabilities.

翻译：大型语言模型（LLM）的发展推动了从浅层指令跟随到严格长程推理的范式转变。尽管组相对策略优化（GRPO）因其卓越性能已成为激发此类训练后推理能力的关键机制，但其仍受制于显著的训练不稳定性和较差的样本效率。我们从理论上将这些问题根源归结为在策略样本中缺乏区分性：对于常规查询，高度同质化的样本会引发破坏性的梯度冲突；而对于困难查询，有效正样本的稀缺则导致优化失效。为弥补这一差距，我们提出了区分性感知的组相对策略优化（DaGRPO）。DaGRPO包含两个核心机制：（1）序列级梯度校正：利用细粒度评分动态屏蔽区分度低的样本对，从而从源头消除梯度冲突；（2）离策略数据增强：引入高质量锚点以恢复困难任务的训练信号。在9个数学推理与分布外（OOD）泛化基准上的大量实验表明，DaGRPO显著超越了现有的监督微调（SFT）、GRPO及混合基线方法，实现了新的最先进性能（例如在数学基准上平均准确率提升+4.7%）。此外，深入分析证实DaGRPO能有效缓解梯度爆炸，并加速长链推理能力的涌现。