Preference learning is a key technology for aligning language models with human values. Reinforcement Learning from Human Feedback (RLHF) is a model-based algorithm to optimize preference learning, which first fits a reward model for preference scores and then optimizes the generating policy with an on-policy PPO algorithm to maximize the reward. The processing of RLHF is complex, time-consuming, and unstable. The Direct Preference Optimization (DPO) algorithm uses an off-policy algorithm to directly optimize the generating policy and eliminates the need for a reward model. DPO is more data-efficient and stable. However, DPO has a drawback of overfitting to the preference data and ignoring the KL-regularization term when the preference is deterministic. Identity mapping Preference Optimization(IPO) uses a root-finding MSE loss to incorporate KL-regularization. However, both DPO and IPO fail to properly address the KL-regularization term because the support of the preference distribution is not equal to the reference distribution. In this paper, we propose a simple and intuitive off-policy preference optimization algorithm from an importance sampling view, which we call Maximum Preference Optimization (MPO). MPO incorporates the off-policy KL-regularization term, making regularization truly effective. MPO achieves the best of both worlds by combining the objectives of RLHF and IPO while being an off-policy algorithm. Furthermore, MPO eliminates the need for a reward model and reference policy, simplifying the learning process and reducing memory usage.

翻译：偏好学习是使语言模型与人类价值观对齐的关键技术。基于人类反馈的强化学习（RLHF）是一种基于模型的偏好学习优化算法，它首先拟合一个奖励模型来评估偏好得分，然后采用同策略PPO算法优化生成策略以最大化奖励。RLHF的处理过程复杂、耗时且不稳定。直接偏好优化（DPO）算法采用异策略算法直接优化生成策略，无需奖励模型，具有更高的数据效率和稳定性。然而，DPO存在一个缺陷：当偏好具有确定性时，它会过度拟合偏好数据而忽略KL正则化项。恒等映射偏好优化（IPO）利用基于根求解的均方误差损失来引入KL正则化。但DPO和IPO均未能妥善处理KL正则化项，原因在于偏好分布的支持集与参考分布不同。本文从重要性采样视角提出一种简单直观的异策略偏好优化算法，称为最大偏好优化（MPO）。MPO引入了异策略KL正则化项，使正则化真正生效。该算法在保持异策略特性的同时，融合RLHF与IPO的目标优势，实现了两类方法的有机统一。此外，MPO无需奖励模型和参考策略，简化了学习过程并降低了内存占用。