While recent preference alignment algorithms for language models have demonstrated promising results, supervised fine-tuning (SFT) remains imperative for achieving successful convergence. In this paper, we study the crucial role of SFT within the context of preference alignment, emphasizing that a minor penalty for the disfavored generation style is sufficient for preference-aligned SFT. Building on this foundation, we introduce a straightforward and innovative reference model-free monolithic odds ratio preference optimization algorithm, ORPO, eliminating the necessity for an additional preference alignment phase. We demonstrate, both empirically and theoretically, that the odds ratio is a sensible choice for contrasting favored and disfavored styles during SFT across the diverse sizes from 125M to 7B. Specifically, fine-tuning Phi-2 (2.7B), Llama-2 (7B), and Mistral (7B) with ORPO on the UltraFeedback alone surpasses the performance of state-of-the-art language models with more than 7B and 13B parameters: achieving up to 12.20% on $\text{AlpacaEval}_{2.0}$ and 7.32 in MT-Bench, as shown in Figures 1 and 12. We release code and model checkpoints for Mistral-ORPO-$\alpha$ (7B) and Mistral-ORPO-$\beta$ (7B).

翻译：尽管近期针对语言模型的偏好对齐算法已展现出令人瞩目的成果，但监督式微调（SFT）仍是实现成功收敛的必要条件。本文深入探究SFT在偏好对齐中的关键作用，强调对非偏好生成风格施加适度惩罚即可实现偏好对齐式SFT。基于此发现，我们提出一种简洁创新的无参考模型单峰胜算比偏好优化算法ORPO，彻底消除额外偏好对齐阶段的需求。我们从理论与实证两方面证明，在125M至7B不同参数规模的SFT过程中，采用胜算比对比偏好与非偏好风格是合理选择。具体而言，仅基于UltraFeedback数据集使用ORPO微调Phi-2 (2.7B)、Llama-2 (7B)及Mistral (7B)模型，其性能即超越参数规模超过7B和13B的先进语言模型：在$\text{AlpacaEval}_{2.0}$上最高达12.20%，MT-Bench评分达7.32（见图1与图12）。我们已开源Mistral-ORPO-$\alpha$ (7B)与Mistral-ORPO-$\beta$ (7B)的代码及模型权重。