Improving the alignment of language models with human preferences remains an active research challenge. Previous approaches have primarily utilized Reinforcement Learning from Human Feedback (RLHF) via online RL methods such as Proximal Policy Optimization (PPO). Recently, offline methods such as Sequence Likelihood Calibration (SLiC) and Direct Preference Optimization (DPO) have emerged as attractive alternatives, offering improvements in stability and scalability while maintaining competitive performance. SLiC refines its loss function using sequence pairs sampled from a supervised fine-tuned (SFT) policy, while DPO directly optimizes language models based on preference data, foregoing the need for a separate reward model. However, the maximum likelihood estimator (MLE) of the target optimal policy requires labeled preference pairs sampled from that policy. DPO's lack of a reward model constrains its ability to sample preference pairs from the optimal policy, and SLiC is restricted to sampling preference pairs only from the SFT policy. To address these limitations, we introduce a novel approach called Statistical Rejection Sampling Optimization (RSO) that aims to source preference data from the target optimal policy using rejection sampling, enabling a more accurate estimation of the optimal policy. We also propose a unified framework that enhances the loss functions used in both SLiC and DPO from a preference modeling standpoint. Through extensive experiments across three diverse tasks, we demonstrate that RSO consistently outperforms both SLiC and DPO on evaluations from both Large Language Model (LLM) and human raters.

翻译：提升语言模型与人类偏好的对齐仍是一个活跃的研究挑战。先前的方法主要通过在线强化学习方法（如近端策略优化（PPO））利用来自人类反馈的强化学习（RLHF）。近年来，序列似然校准（SLiC）和直接偏好优化（DPO）等离线方法成为有吸引力的替代方案，在保持竞争性能的同时，提升了稳定性与可扩展性。SLiC使用从监督微调（SFT）策略中采样的序列对来优化其损失函数，而DPO则基于偏好数据直接优化语言模型，无需单独奖励模型。然而，目标最优策略的最大似然估计（MLE）需要从该策略中采样的带标签偏好对。DPO缺乏奖励模型限制其从最优策略中采样偏好对的能力，而SLiC仅限于从SFT策略中采样偏好对。为解决这些局限性，我们提出了一种名为统计拒绝采样优化（RSO）的新方法，旨在利用拒绝采样从目标最优策略中获取偏好数据，从而更精确地估计最优策略。我们还提出了一个统一框架，从偏好建模的角度改进了SLiC和DPO中使用的损失函数。通过在三个不同任务上的大量实验，我们证明RSO在大语言模型（LLM）和人类评估者的评价中均持续优于SLiC和DPO。