Hybrid training methods for large language models combine supervised fine tuning (SFT) on expert demonstrations with reinforcement learning (RL) on model rollouts, typically at the sample level. We propose Entropy Gated Selective Policy Optimization (EGSPO), a three stage framework that extends sample level mixing with token level gradient modulation. Stage 1, SFT expert learning, establishes a reliable warm up policy using expert demonstrations with a pure SFT loss. Stage 2, RL rollout generation, samples trajectories from the current policy and computes per token predictive entropy. Stage 3, the EGSPO mechanism, applies entropy gated gradient allocation: a predictive entropy module routes high entropy tokens to full PPO updates to encourage exploration, and low entropy tokens to attenuated PPO updates to reduce variance and preserve knowledge. Critically, both branches incorporate the advantage function A_t, ensuring that incorrect trajectories receive consistent negative learning signals and preventing reinforcement of confident errors. EGSPO achieves consistent improvements on mathematical reasoning benchmarks, with gains of 3.8 percent on AIME and 2.9 percent on MATH over the CHORD phi baseline, while incurring only 3.4 percent additional computational overhead.

翻译：大语言模型的混合训练方法通常结合了基于专家演示的监督微调（SFT）与基于模型生成序列的强化学习（RL），且通常在样本层面进行混合。我们提出了熵门控选择性策略优化（EGSPO），这是一个三阶段框架，通过在令牌级别进行梯度调制，扩展了样本层面的混合方法。第一阶段，SFT专家学习，使用纯SFT损失在专家演示上建立一个可靠的预热策略。第二阶段，RL生成序列，从当前策略中采样轨迹并计算每个令牌的预测熵。第三阶段，即EGSPO机制，应用熵门控梯度分配：一个预测熵模块将高熵令牌路由至完整的PPO更新以鼓励探索，并将低熵令牌路由至衰减的PPO更新以减少方差并保留知识。关键在于，两个分支都整合了优势函数A_t，确保错误的轨迹接收到一致的负学习信号，从而防止强化自信的错误。EGSPO在数学推理基准测试上取得了持续的改进，相较于CHORD phi基线，在AIME上提升了3.8%，在MATH上提升了2.9%，同时仅产生了3.4%的额外计算开销。