This paper presents Multi-Objective Reinforcement Learning from AI Feedback (MORLAIF), a novel approach to improving the alignment and performance of language models trained using reinforcement learning from AI feedback (RLAIF). In contrast to standard approaches that train a single preference model to represent all human preferences, MORLAIF decomposes this task into multiple simpler principles, such as toxicity, factuality, and sycophancy. Separate preference models are trained for each principle using feedback from GPT-3.5-Turbo. These preference model scores are then combined using different scalarization functions to provide a reward signal for Proximal Policy Optimization (PPO) training of the target language model. Our experiments indicate that MORLAIF outperforms the standard RLAIF baselines and that MORLAIF can be used to align larger language models using smaller ones. Surprisingly, the choice of scalarization function does not appear to significantly impact the results.

翻译：本文提出了一种新颖的方法——基于人工智能反馈的多目标强化学习（MORLAIF），旨在提升通过人工智能反馈强化学习（RLAIF）训练的语言模型的对齐性与性能。与训练单一偏好模型以代表所有人类偏好的标准方法不同，MORLAIF将此任务分解为多个更简单的原则，例如毒性、事实性和谄媚性。我们利用GPT-3.5-Turbo的反馈，为每个原则分别训练了独立的偏好模型。随后，通过不同的标量化函数将这些偏好模型的分数进行组合，为目标语言模型的近端策略优化（PPO）训练提供奖励信号。实验结果表明，MORLAIF的表现优于标准的RLAIF基线方法，并且能够利用较小的语言模型来对齐较大的语言模型。出乎意料的是，标量化函数的选择似乎对结果没有显著影响。