While recent preference alignment algorithms for language models have demonstrated promising results, supervised fine-tuning (SFT) remains imperative for achieving successful convergence. In this paper, we study the crucial role of SFT within the context of preference alignment, emphasizing that a minor penalty for the disfavored generation style is sufficient for preference-aligned SFT. Building on this foundation, we introduce a straightforward and innovative reference model-free monolithic odds ratio preference optimization algorithm, ORPO, eliminating the necessity for an additional preference alignment phase. We demonstrate, both empirically and theoretically, that the odds ratio is a sensible choice for contrasting favored and disfavored styles during SFT across the diverse sizes from 125M to 7B. Specifically, fine-tuning Phi-2 (2.7B), Llama-2 (7B), and Mistral (7B) with ORPO on the UltraFeedback alone surpasses the performance of state-of-the-art language models with more than 7B and 13B parameters: achieving up to 12.20% on $\text{AlpacaEval}_{2.0}$ (Figure 1), 66.19% on IFEval (instruction-level loose, Table 6), and 7.32 in MT-Bench (Figure 12). We release code and model checkpoints for Mistral-ORPO-$\alpha$ (7B) and Mistral-ORPO-$\beta$ (7B).

翻译：尽管近期针对语言模型的偏好对齐算法已展现良好效果，监督微调（SFT）仍是实现成功收敛的必要步骤。本文深入研究了SFT在偏好对齐中的关键作用，强调对不偏好生成风格施加轻微惩罚即足以实现偏好对齐的SFT。基于这一发现，我们提出了一种简洁创新的免参考模型单阶段比值比偏好优化算法ORPO，消除了额外偏好对齐阶段的必要性。我们从理论与实证两个层面证明，在125M至7B参数规模的SFT过程中，比值比是区分偏好与非偏好风格的有效选择。具体而言，单独使用UltraFeedback数据集对Phi-2（2.7B）、Llama-2（7B）及Mistral（7B）进行ORPO微调后，其性能超越了多数拥有7B与13B参数的先进语言模型：在$\text{AlpacaEval}_{2.0}$上达到12.20%（图1），在IFEval（指令级宽松评估，表6）上达到66.19%，在MT-Bench上达到7.32分（图12）。我们已发布Mistral-ORPO-$\alpha$（7B）及Mistral-ORPO-$\beta$（7B）的代码与模型检查点。