Optimization (PPO) has been positioned by recent literature as the canonical method for the RL part of RLHF. PPO performs well empirically but has a heuristic motivation and handles the KL-divergence constraint used in LM-RLHF in an ad-hoc manner and suffers form reward oscillations, entropy collapse, value function drift, and sudden policy divergence that require frequent restarts and extensive hyperparameter tuning. In this paper, we develop a new pure on policy actor-critic RL method for the LM-RLHF setting. We present SAFE (Stable Alignment Finetuning with Entropy-aware control),a novel RLHF algorithm that combines a Double Soft-Min Critic for pessimistic value estimation with a new multi-layer stabilization framework combining entropy-gated KL regulation, and PID-controlled adaptive thresholds. Unlike standard PPO's symmetric KL penalties, SAFE distinguishes high-entropy exploration from low-entropy mode collapse and adjusts penalties dynamically based on reward velocity. Experiments on a 3B parameter model show SAFE achieves +5.15\% training-average reward than PPO (0.725 vs 0.689), negligible reward crashes, and superior KL control than ppo . Our method adds minimal computational overhead and provides an interpretable, crash-resistant RLHF framework that maintains aggressive learning speed while ensuring stable long-horizon optimization suitable for production deployment. Code is available at https://github.com/ryyzn9/SAFE

翻译：近期文献将近端策略优化（PPO）定位为强化学习人类反馈（RLHF）中强化学习部分的标准方法。PPO在实证中表现良好，但其动机具有启发性，并以临时方式处理语言模型强化学习人类反馈（LM-RLHF）中使用的KL散度约束，且存在奖励振荡、熵崩溃、价值函数漂移以及需要频繁重启和大量超参数调整的突发策略发散问题。本文针对LM-RLHF场景提出一种全新的纯在线策略执行者-评论者强化学习方法。我们提出了SAFE（基于熵感知控制的稳定对齐微调）——一种新型RLHF算法，该算法融合了用于悲观价值估计的双重软最小评论者，以及结合熵门控KL调节与PID控制自适应阈值的新型多层稳定框架。与标准PPO的对称KL惩罚不同，SAFE能够区分高熵探索与低熵模式崩溃，并根据奖励变化速度动态调整惩罚力度。在30亿参数模型上的实验表明，SAFE相比PPO实现了+5.15%的训练平均奖励提升（0.725 vs 0.689），奖励崩溃可忽略不计，且KL控制能力优于PPO。本方法仅增加极少的计算开销，提供了一个可解释、抗崩溃的RLHF框架，在保持激进学习速度的同时，确保适用于生产部署的稳定长程优化。代码发布于https://github.com/ryyzn9/SAFE