In aligning large language models (LLMs), reward models have played an important role, but are standardly trained as discriminative models and rely only on labeled human preference data. In this paper, we explore methods that train reward models using both unlabeled and labeled data. Building on the generative models in LLMs, we develop a generative reward model that is first trained via large-scale unsupervised learning and then fine-tuned via supervised learning. We also show that by using label smoothing, we are in fact optimizing a regularized pairwise ranking loss. This result, in turn, provides a new view of training reward models, which links generative models and discriminative models under the same class of training objectives. The outcome of these techniques is a foundation reward model, which can be applied to a wide range of tasks with little or no further fine-tuning effort. Extensive experiments show that this model generalizes well across several tasks, including response ranking, reinforcement learning from human feedback, and task adaptation with fine-tuning, achieving significant performance improvements over several strong baseline models.

翻译：在对齐大型语言模型（LLMs）的过程中，奖励模型发挥着重要作用，但传统方法通常将其训练为判别式模型，且仅依赖标注的人类偏好数据。本文探索了利用未标注和已标注数据联合训练奖励模型的方法。基于LLMs中的生成式模型架构，我们开发了一种生成式奖励模型：首先通过大规模无监督学习进行预训练，随后通过监督学习进行微调。我们还证明，通过使用标签平滑技术，实际上是在优化一种正则化的成对排序损失。这一结果反过来为训练奖励模型提供了新的视角，将生成式模型与判别式模型在同一类训练目标下联系起来。这些技术最终产生了一个基础奖励模型，该模型可广泛应用于多种任务，且无需或仅需极少额外微调。大量实验表明，该模型在多项任务中均表现出良好的泛化能力，包括响应排序、基于人类反馈的强化学习以及微调任务适应，相较于多个强基线模型取得了显著的性能提升。