Aligning language models with human preferences is essential for ensuring their safety and reliability. Although most existing approaches assume specific human preference models such as the Bradley-Terry model, this assumption may fail to accurately capture true human preferences, and consequently, these methods lack statistical consistency, i.e., the guarantee that language models converge to the true human preference as the number of samples increases. In contrast, direct density ratio optimization (DDRO) achieves statistical consistency without assuming any human preference models. DDRO models the density ratio between preferred and non-preferred data distributions using the language model, and then optimizes it via density ratio estimation. However, this density ratio is unstable and often diverges, leading to training instability of DDRO. In this paper, we propose a novel alignment method that is both stable and statistically consistent. Our approach is based on the relative density ratio between the preferred data distribution and a mixture of the preferred and non-preferred data distributions. Our approach is stable since this relative density ratio is bounded above and does not diverge. Moreover, it is statistically consistent and yields significantly tighter convergence guarantees than DDRO. We experimentally show its effectiveness with Qwen 2.5 and Llama 3.

翻译：对齐语言模型与人类偏好对于确保其安全性和可靠性至关重要。尽管现有方法大多假设特定的人类偏好模型（如Bradley-Terry模型），但这一假设可能无法准确捕捉真实的人类偏好，因此这些方法缺乏统计一致性，即随着样本数量增加，语言模型无法保证收敛到真实的人类偏好。相比之下，直接密度比优化（DDRO）无需假设任何人类偏好模型即可实现统计一致性。DDRO利用语言模型对偏好数据与非偏好数据分布之间的密度比进行建模，并通过密度比估计对其进行优化。然而，该密度比不稳定且常常发散，导致DDRO的训练不稳定。本文提出了一种既稳定又具备统计一致性的新型对齐方法。我们的方法基于偏好数据分布与偏好及非偏好数据混合分布之间的相对密度比。由于该相对密度比有上界且不发散，因此方法稳定。此外，该方法具有统计一致性，且收敛性保证显著优于DDRO。我们通过Qwen 2.5和Llama 3实验验证了其有效性。