Language model (LM) post-training (or alignment) involves maximizing a reward function that is derived from preference annotations. Direct Preference Optimization (DPO) is a popular offline alignment method that trains a policy directly on preference data without the need to train a reward model or apply reinforcement learning. However, typical preference datasets have only a single, or at most a few, annotation per preference pair, which causes DPO to overconfidently assign rewards that trend towards infinite magnitude. This frequently leads to degenerate policies, sometimes causing even the probabilities of the preferred generations to go to zero. In this work, we analyze this phenomenon and propose distillation to get a better proxy for the true preference distribution over generation pairs: we train the LM to produce probabilities that match the distribution induced by a reward model trained on the preference data. Moreover, to account for uncertainty in the reward model we are distilling from, we optimize against a family of reward models that, as a whole, is likely to include at least one reasonable proxy for the preference distribution. Our results show that distilling from such a family of reward models leads to improved robustness to distribution shift in preference annotations, while preserving the simple supervised nature of DPO.

翻译：语言模型（LM）的后训练（或对齐）过程涉及最大化基于偏好标注推导出的奖励函数。直接偏好优化（DPO）是一种流行的离线对齐方法，它直接在偏好数据上训练策略，无需训练奖励模型或应用强化学习。然而，典型的偏好数据集对每个偏好对仅包含单个（或最多数个）标注，这导致DPPO会过度自信地分配趋向无穷大的奖励值。这种现象经常引发策略退化，有时甚至会使优选生成的概率降至零。本研究分析了该现象，并提出通过蒸馏方法获得生成对真实偏好分布的更优代理：我们训练语言模型使其输出的概率分布与基于偏好数据训练的奖励模型所诱导的分布相匹配。此外，为处理被蒸馏奖励模型的不确定性，我们针对一个奖励模型族进行优化，该模型族整体上很可能包含至少一个对偏好分布的合理代理。实验结果表明，从此类奖励模型族进行蒸馏能提升对偏好标注分布偏移的鲁棒性，同时保持DPPO简单的监督学习特性。