Offline preference optimization allows fine-tuning large models directly from offline data, and has proved effective in recent alignment practices. We propose generalized preference optimization (GPO), a family of offline losses parameterized by a general class of convex functions. GPO enables a unified view over preference optimization, encompassing existing algorithms such as DPO, IPO and SLiC as special cases, while naturally introducing new variants. The GPO framework also sheds light on how offline algorithms enforce regularization, through the design of the convex function that defines the loss. Our analysis and experiments reveal the connections and subtle differences between the offline regularization and the KL divergence regularization intended by the canonical RLHF formulation. In all, our results present new algorithmic toolkits and empirical insights to alignment practitioners.

翻译：离线偏好优化允许直接从离线数据中微调大型模型，并在近期的对齐实践中证明了有效性。我们提出广义偏好优化（GPO），这是一类由一般凸函数参数化的离线损失函数族。GPO实现了对偏好优化的统一视角，将现有算法（如DPO、IPO和SLiC）作为特例纳入其中，同时自然衍生出新的变体。GPO框架还通过定义损失的凸函数设计，揭示了离线算法如何强制执行正则化。我们的分析与实验揭示了离线正则化与经典RLHF公式所预期的KL散度正则化之间的关联与微妙差异。总体而言，我们的结果为对齐实践者提供了新的算法工具集与实证见解。