Large language models are typically aligned with human preferences by optimizing $\textit{reward models}$ (RMs) fitted to human feedback. However, human preferences are multi-faceted, and it is increasingly common to derive reward from a composition of simpler reward models which each capture a different aspect of language quality. This itself presents a challenge, as it is difficult to appropriately weight these component RMs when combining them. Compounding this difficulty, because any RM is only a proxy for human evaluation, this process is vulnerable to $\textit{overoptimization}$, wherein past a certain point, accumulating higher reward is associated with worse human ratings. In this paper, we perform, to our knowledge, the first study on overoptimization in composite RMs, showing that correlation between component RMs has a significant effect on the locations of these points. We then introduce an approach to solve this issue using constrained reinforcement learning as a means of preventing the agent from exceeding each RM's threshold of usefulness. Our method addresses the problem of weighting component RMs by learning dynamic weights, naturally given by the Lagrange multipliers. As a result, each RM stays within the range at which it is an effective proxy, improving evaluation performance. Finally, we introduce an adaptive method using gradient-free optimization to identify and optimize towards these points during a single run.

翻译：大型语言模型通常通过优化基于人类反馈拟合的 $\textit{奖励模型}$ 与人类偏好对齐。然而，人类偏好是多维度的，通过组合多个分别捕捉语言质量不同方面的简单奖励模型来推导奖励的做法日益普遍。这本身带来了挑战，因为组合这些子奖励模型时难以适当加权。雪上加霜的是，由于任何奖励模型都仅是人类评估的代理，这一过程容易遭受 $\textit{过优化}$，即超过某一点后，累积更高奖励反而对应更差的人类评估。本文首次对复合奖励模型中的过优化进行研究，表明子奖励模型间的相关性对这些点的位置具有显著影响。我们随后提出一种基于约束强化学习的解决方案，通过限制智能体不超过每个奖励模型的有效阈值来应对该问题。该方法通过学习动态权重（由拉格朗日乘子自然给出）解决了子奖励模型的加权难题。由此，每个奖励模型都能保持在作为有效代理的范围内，提升评估性能。最后，我们引入一种基于无梯度优化的自适应方法，可在单次运行中识别并优化这些临界点。