Preference learning algorithms (e.g., RLHF and DPO) are frequently used to steer LLMs to produce generations that are more preferred by humans, but our understanding of their inner workings is still limited. In this work, we study the conventional wisdom that preference learning trains models to assign higher likelihoods to more preferred outputs than less preferred outputs, measured via ranking accuracy. Surprisingly, we find that most state-of-the-art preference-tuned models achieve a ranking accuracy of less than 60% on common preference datasets. We furthermore derive the idealized ranking accuracy that a preference-tuned LLM would achieve if it optimized the DPO or RLHF objective perfectly. We demonstrate that existing models exhibit a significant alignment gap -- i.e., a gap between the observed and idealized ranking accuracies. We attribute this discrepancy to the DPO objective, which is empirically and theoretically ill-suited to fix even mild ranking errors in the reference model, and derive a simple and efficient formula for quantifying the difficulty of learning a given preference datapoint. Finally, we demonstrate that ranking accuracy strongly correlates with the empirically popular win rate metric when the model is close to the reference model used in the objective, shedding further light on the differences between on-policy (e.g., RLHF) and off-policy (e.g., DPO) preference learning algorithms.

翻译：偏好学习算法（例如RLHF和DPO）常被用于引导大语言模型生成更受人类偏好的内容，但我们对其内部工作机制的理解仍然有限。在本研究中，我们探讨了一个传统观点：偏好学习训练模型为更受偏好的输出分配比次偏好输出更高的似然度，这一能力通过排序准确率来衡量。令人惊讶的是，我们发现大多数先进的偏好调优模型在常见偏好数据集上的排序准确率低于60%。此外，我们推导了若偏好调优的大语言模型能完美优化DPO或RLHF目标时所应达到的理想化排序准确率。我们证明现有模型存在显著的"对齐鸿沟"——即观测排序准确率与理想化排序准确率之间的差距。我们将这种差异归因于DPO目标函数，该函数在实证和理论上均难以修正参考模型中即使轻微的排序错误，并推导出一个量化给定偏好数据点学习难度的简洁高效公式。最后，我们证明当模型接近目标函数中使用的参考模型时，排序准确率与当前流行的胜率指标呈现强相关性，这进一步揭示了在线策略（如RLHF）与离线策略（如DPO）偏好学习算法之间的差异。