Great successes have been reported using Reinforcement Learning from Human Feedback (RLHF) to align large language models. Open-source preference datasets and reward models have enabled wider experimentation beyond generic chat settings, particularly to make systems more "helpful" for tasks like web question answering, summarization, and multi-turn dialogue. When optimizing for helpfulness, RLHF has been consistently observed to drive models to produce longer outputs. This paper demonstrates that optimizing for response length is a significant factor behind RLHF's reported improvements in these settings. First, we study the relationship between reward and length for reward models trained on three open-source preference datasets for helpfulness. Here, length correlates strongly with reward, and improvements in reward score are driven in large part by shifting the distribution over output lengths. We then explore interventions during both RL and reward model learning to see if we can achieve the same downstream improvements as RLHF without increasing length. While our interventions mitigate length increases, they aren't uniformly effective across settings. Furthermore, we find that even running RLHF with a reward based solely on length can reproduce most of the downstream improvements over the initial policy model, showing that reward models in these settings have a long way to go.

翻译：利用人类反馈强化学习对齐大型语言模型已报告取得重大成功。开源偏好数据集和奖励模型使得在通用聊天场景之外的更广泛实验成为可能，特别是使系统在网页问答、摘要和多轮对话等任务中更具"有用性"。在优化有用性时，RLHF一直被发现会促使模型生成更长的输出。本文证明，优化响应长度是RLHF在这些场景下报告性能提升的关键因素。首先，我们研究了针对三个开源偏好数据集训练的有用性奖励模型中奖励与长度之间的关系。在此，长度与奖励高度相关，奖励分数的提升主要源于输出长度分布的偏移。随后，我们探索了在强化学习和奖励模型学习中的干预措施，以考察能否在不增加长度的情况下实现与RLHF相同的下游改进。尽管我们的干预措施缓解了长度增加，但在不同场景下效果并不一致。此外，我们发现即使仅基于长度的奖励运行RLHF，也能复现大多数相对于初始策略模型的下游改进，这表明这些场景中的奖励模型还有很长的路要走。