Self-reflection capabilities emerge in Large Language Models after RL post-training, with multi-turn RL achieving substantial gains over SFT counterparts. Yet the mechanism of how a unified optimization objective gives rise to functionally distinct capabilities of generating solutions and evaluating when to revise them remains opaque. To address this question, we introduce the Gradient Attribution Property to characterize how reward gradients distribute across policy components, formalized through the Two-Stage Decision-Sampling (DS) Hypothesis, which decomposes the policy into sampling ($π_{sample}$) for generation and decision ($π_{d}$) for verification. We prove that surrogate rewards exhibit Balanced Gradient Attribution, while SFT and KL penalties exhibit Unbalanced Gradient Attribution, with length-weighting creating asymmetric regularization that constrains $π_{sample}$ while leaving $π_{d}$ under-optimized, providing an theoretical explanation of why RL succeeds where SFT fails. We also empirically validate our theoretical predictions on arithmetic reasoning demonstrates that RL's superior generalization stems primarily from improved decision-making ($π_{d}$) rather than sampling capabilities, providing a first-principles mechanistic explanation for self-correction in thinking models.

翻译：自我反思能力在大语言模型经过强化学习（RL）后训练后涌现，其中多轮RL相比监督微调（SFT）模型取得了显著提升。然而，统一的优化目标如何催生生成解决方案与评估何时修订方案这两种功能上截然不同的能力，其机制仍不明确。为探究此问题，我们引入梯度归因特性来描述奖励梯度在策略组件间的分布规律，并通过两阶段决策-采样（DS）假说形式化该特性——该假说将策略分解为用于生成的采样策略（$π_{sample}$）和用于验证的决策策略（$π_{d}$）。我们证明：代理奖励呈现平衡梯度归因特性，而SFT与KL惩罚则呈现非平衡梯度归因特性；其中长度加权机制形成的不对称正则化会约束$π_{sample}$，同时导致$π_{d}$优化不足，这从理论上解释了为何RL在SFT失败的场景中能够成功。我们进一步在算术推理任务上实证验证了理论预测，表明RL的卓越泛化能力主要源于决策能力（$π_{d}$）的提升而非采样能力，从而为思维模型中的自我修正机制提供了基于第一性原理的机理性解释。