The extraordinary capabilities of large language models (LLMs) such as ChatGPT and GPT-4 are in part unleashed by aligning them with reward models that are trained on human preferences, which are often represented as rankings of responses to prompts. In this paper, we document the phenomenon of \textit{reward collapse}, an empirical observation where the prevailing ranking-based approach results in an \textit{identical} reward distribution \textit{regardless} of the prompts during the terminal phase of training. This outcome is undesirable as open-ended prompts like ``write a short story about your best friend'' should yield a continuous range of rewards for their completions, while specific prompts like ``what is the capital of New Zealand'' should generate either high or low rewards. Our theoretical investigation reveals that reward collapse is primarily due to the insufficiency of the ranking-based objective function to incorporate prompt-related information during optimization. This insight allows us to derive closed-form expressions for the reward distribution associated with a set of utility functions in an asymptotic regime. To overcome reward collapse, we introduce a prompt-aware optimization scheme that provably admits a prompt-dependent reward distribution within the interpolating regime. Our experimental results suggest that our proposed prompt-aware utility functions significantly alleviate reward collapse during the training of reward models.

翻译：大语言模型（如ChatGPT和GPT-4）的卓越能力部分源于其通过与基于人类偏好训练的奖励模型对齐而释放，这些偏好通常以提示响应的排序形式表示。本文记录了“奖励坍缩”现象，即一种实证观察结果：在训练终止阶段，基于排序的常用方法会导致奖励分布与提示内容无关，呈现完全相同的形式。这一结果是不理想的，因为诸如“写一篇关于你最好的朋友的短故事”这样的开放式提示应为其生成结果产生连续的奖励区间，而诸如“新西兰的首都是哪里”这样的具体提示则应生成高或低奖励。我们的理论研究表明，奖励坍缩的主要原因是基于排序的目标函数在优化过程中未能充分整合提示相关信息。这一见解使我们能够在渐近状态下推导出与一组效用函数相关的奖励分布的闭式表达式。为解决奖励坍缩，我们提出了一种提示感知优化方案，该方案在插值状态下可证明地产生与提示相关的奖励分布。实验结果表明，我们提出的提示感知效用函数在奖励模型训练过程中显著缓解了奖励坍缩现象。