While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward without drifting too far from the original model. In this paper, we leverage a mapping between reward functions and optimal policies to show that this constrained reward maximization problem can be optimized exactly with a single stage of policy training, essentially solving a classification problem on the human preference data. The resulting algorithm, which we call Direct Preference Optimization (DPO), is stable, performant and computationally lightweight, eliminating the need for fitting a reward model, sampling from the LM during fine-tuning, or performing significant hyperparameter tuning. Our experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods. Notably, fine-tuning with DPO exceeds RLHF's ability to control sentiment of generations and improves response quality in summarization and single-turn dialogue while being substantially simpler to implement and train.

翻译：尽管大规模无监督语言模型（LM）能学习广泛的世界知识和一定的推理能力，但由于其完全无监督的训练性质，难以实现对其行为的精确控制。现有实现这种可控性的方法会收集关于模型生成结果相对质量的人工标注，并通过微调无监督语言模型使其与这些偏好对齐，通常采用基于人类反馈的强化学习（RLHF）。然而，RLHF是一个复杂且常不稳定的过程：首先拟合一个反映人类偏好的奖励模型，然后利用强化学习微调大规模无监督语言模型，以最大化这一估计奖励，同时避免偏离原始模型过远。本文利用奖励函数与最优策略之间的映射关系，证明了这一带约束的奖励最大化问题可通过单阶段策略训练精确优化，本质上是在人类偏好数据上求解一个分类问题。由此产生的算法——我们称之为直接偏好优化（DPO）——具有稳定性、高性能和计算轻量的特点，无需拟合奖励模型、在微调过程中对语言模型进行采样，或进行大量超参数调优。实验表明，DPO在微调语言模型以对齐人类偏好方面，效果可与现有方法相当甚至更优。值得注意的是，使用DPO进行微调在情感控制能力上超越了RLHF，并在摘要生成和单轮对话中提升了响应质量，同时实现和训练过程均显著简化。