While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward without drifting too far from the original model. In this paper we introduce a new parameterization of the reward model in RLHF that enables extraction of the corresponding optimal policy in closed form, allowing us to solve the standard RLHF problem with only a simple classification loss. The resulting algorithm, which we call Direct Preference Optimization (DPO), is stable, performant, and computationally lightweight, eliminating the need for sampling from the LM during fine-tuning or performing significant hyperparameter tuning. Our experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods. Notably, fine-tuning with DPO exceeds PPO-based RLHF in ability to control sentiment of generations, and matches or improves response quality in summarization and single-turn dialogue while being substantially simpler to implement and train.

翻译：尽管大规模无监督语言模型能够学习广泛的世界知识和部分推理能力，但由于其完全无监督的训练特性，很难实现对模型行为的精确控制。现有方法通过收集人类对模型生成内容相对质量的标注，并微调无监督语言模型以对齐这些偏好（通常采用基于人类反馈的强化学习）。然而，RLHF是一个复杂且不稳定的过程：首先需要拟合反映人类偏好的奖励模型，再通过强化学习微调大型无监督语言模型，使其在最大化预估奖励的同时避免偏离原始模型过远。本文提出一种RLHF奖励模型的新参数化方法，能够以封闭形式直接推导出对应的最优策略，从而仅需通过简单的分类损失函数即可解决标准RLHF问题。由此产生的算法——直接偏好优化——具有稳定性强、性能优异且计算量轻的特点，在微调过程中既无需从语言模型采样，也无需进行大量超参数调整。实验表明，DPO对语言模型进行人类偏好对齐微调的效果可与现有方法媲美甚至更优。值得注意的是，采用DPO微调在生成情感控制能力上超越基于PPO的RLHF，在摘要生成与单轮对话任务中维持或提升回复质量，同时实现和训练流程却显著简化。