Reinforcement Learning from Human Feedback (RLHF) has become a crucial technology for aligning language models with human values and intentions, enabling models to produce more helpful and harmless responses. Reward models are trained as proxies for human preferences to drive reinforcement learning optimization. While reward models are often considered central to achieving high performance, they face the following challenges in practical applications: (1) Incorrect and ambiguous preference pairs in the dataset may hinder the reward model from accurately capturing human intent. (2) Reward models trained on data from a specific distribution often struggle to generalize to examples outside that distribution and are not suitable for iterative RLHF training. In this report, we attempt to address these two issues. (1) From a data perspective, we propose a method to measure the strength of preferences within the data, based on a voting mechanism of multiple reward models. Experimental results confirm that data with varying preference strengths have different impacts on reward model performance. We introduce a series of novel methods to mitigate the influence of incorrect and ambiguous preferences in the dataset and fully leverage high-quality preference data. (2) From an algorithmic standpoint, we introduce contrastive learning to enhance the ability of reward models to distinguish between chosen and rejected responses, thereby improving model generalization. Furthermore, we employ meta-learning to enable the reward model to maintain the ability to differentiate subtle differences in out-of-distribution samples, and this approach can be utilized for iterative RLHF optimization.

翻译：基于人类反馈的强化学习（RLHF）已成为将语言模型与人类价值观和意图对齐的关键技术，使模型能够生成更实用且无害的响应。奖励模型作为人类偏好的代理进行训练，以驱动强化学习优化。尽管奖励模型常被视为实现高性能的核心，但在实际应用中面临以下挑战：（1）数据集中的错误与模糊偏好对可能阻碍奖励模型准确捕捉人类意图；（2）在特定分布数据上训练的奖励模型往往难以泛化至分布外样本，且不适用于迭代式RLHF训练。本报告尝试解决这两个问题：（1）从数据角度，我们提出基于多奖励模型投票机制的方法来衡量数据中的偏好强度。实验结果证实，不同偏好强度的数据对奖励模型性能具有差异化影响。我们引入一系列创新方法，以减轻数据集中错误与模糊偏好的影响，并充分利用高质量偏好数据。（2）从算法角度，我们引入对比学习增强奖励模型区分选定与拒绝响应的能力，进而提升模型泛化性。此外，我们采用元学习使奖励模型在分布外样本中仍能保持对细微差异的辨识力，该方法可应用于迭代式RLHF优化。