Pretrained language models are commonly aligned with human preferences and downstream tasks via reinforcement finetuning (RFT), which entails maximizing a (possibly learned) reward function using policy gradient algorithms. This work highlights a fundamental optimization obstacle in RFT: we prove that the expected gradient for an input vanishes when its reward standard deviation under the model is small, even if the expected reward is far from optimal. Through experiments on an RFT benchmark and controlled environments, as well as a theoretical analysis, we then demonstrate that vanishing gradients due to small reward standard deviation are prevalent and detrimental, leading to extremely slow reward maximization. Lastly, we explore ways to overcome vanishing gradients in RFT. We find the common practice of an initial supervised finetuning (SFT) phase to be the most promising candidate, which sheds light on its importance in an RFT pipeline. Moreover, we show that a relatively small number of SFT optimization steps on as few as 1% of the input samples can suffice, indicating that the initial SFT phase need not be expensive in terms of compute and data labeling efforts. Overall, our results emphasize that being mindful for inputs whose expected gradient vanishes, as measured by the reward standard deviation, is crucial for successful execution of RFT.

翻译：预训练语言模型通常通过强化微调（RFT）与人类偏好及下游任务对齐，其核心是利用策略梯度算法最大化（可能经过学习的）奖励函数。本研究揭示了RFT中一个根本性的优化障碍：我们证明，当输入在模型下的奖励标准差较小时，其期望梯度会消失，即便期望奖励远未达到最优。通过在RFT基准测试、受控环境中的实验以及理论分析，我们进一步论证了由小奖励标准差导致的梯度消失现象普遍存在且危害显著，会引发极其缓慢的奖励最大化过程。最后，我们探索了克服RFT中梯度消失问题的可行方法。研究发现，初始监督微调（SFT）阶段作为常用实践是最具潜力的缓解方案，揭示了其在RFT流程中的关键作用。此外，我们证明仅需对1%的输入样本进行少量SFT优化步骤即可满足要求，表明初始SFT阶段在计算和数据标注成本上无需过高。总体而言，我们的研究结果强调：通过奖励标准差衡量输入样本的期望梯度是否消失，是成功实施RFT的核心要素。