Pretrained language models are commonly aligned with human preferences and downstream tasks via reinforcement finetuning (RFT), which refers to maximizing a (possibly learned) reward function using policy gradient algorithms. This work identifies a fundamental optimization obstacle in RFT: we prove that the expected gradient for an input vanishes when its reward standard deviation under the model is small, even if the expected reward is far from optimal. Through experiments on an RFT benchmark and controlled environments, as well as a theoretical analysis, we then demonstrate that vanishing gradients due to small reward standard deviation are prevalent and detrimental, leading to extremely slow reward maximization. Lastly, we explore ways to overcome vanishing gradients in RFT. We find the common practice of an initial supervised finetuning (SFT) phase to be the most promising candidate, which sheds light on its importance in an RFT pipeline. Moreover, we show that a relatively small number of SFT optimization steps on as few as 1% of the input samples can suffice, indicating that the initial SFT phase need not be expensive in terms of compute and data labeling efforts. Overall, our results emphasize that being mindful for inputs whose expected gradient vanishes, as measured by the reward standard deviation, is crucial for successful execution of RFT.

翻译：预训练语言模型通常通过强化微调（RFT）与人类偏好及下游任务对齐，该过程利用策略梯度算法最大化（可能为学习得到的）奖励函数。本研究揭示了RFT中的根本性优化障碍：我们证明当输入在模型下的奖励标准差较小时，其期望梯度会消失，即使期望奖励远非最优。通过在RFT基准测试及受控环境下的实验，结合理论分析，我们进一步验证了小奖励标准差导致的梯度消失现象普遍存在且具有危害性，会引发极其缓慢的奖励最大化进程。最后，我们探索了克服RFT中梯度消失的方法。发现初始监督微调（SFT）阶段是最有前景的候选方案，这揭示了其在RFT流程中的重要性。此外，我们证明在仅1%的输入样本上进行少量SFT优化步骤即可奏效，表明初始SFT阶段无需消耗大量计算与数据标注资源。总体而言，我们的研究结果强调，关注输入中奖励标准差所反映的期望梯度消失现象，对于成功执行RFT至关重要。