Recent alignment methods based on Direct Preference Optimization (DPO) reformulate preference learning as supervised optimization over pairwise comparisons, offering improved efficiency and stability over reinforcement learning from human feedback (RLHF). However, existing DPO-style methods implicitly assume a single fixed preference objective, which limits their ability to model the structured and sometimes conflicting nature of real-world human judgments that span multiple preference dimensions. In this work, we propose Listwise Direct Preference Optimization ($λ$-DPO), a unified framework that simultaneously improves supervision granularity and preference flexibility. Instead of collapsing multi-dimensional preference signals into a single ranking, $λ$-DPO constructs a mixture of listwise preference distributions weighted by a preference vector $λ$ on the probability simplex, enabling a single model to internalize a continuous spectrum of preference trade-offs. To further improve robustness, we introduce a performance-driven stochastic $λ$ scheduler that adaptively samples preference weights based on empirical downstream performance, explicitly mitigating the risks of misspecification inherent to static weighting schemes. We evaluate our method across multiple model families and scales on six widely used benchmarks. Experimental results show the consistent improvement against baselines.

翻译：基于直接偏好优化（DPO）的最新对齐方法将偏好学习重构为基于成对比较的监督优化，相比基于人类反馈的强化学习（RLHF）提高了效率与稳定性。然而，现有的DPO类方法隐含地假设了单一固定的偏好目标，这限制了其对现实世界人类判断的建模能力——这些判断通常具有结构化特征，且可能跨越多个偏好维度并存在内在冲突。本文提出列表式直接偏好优化（$λ$-DPO），这是一个能够同时提升监督粒度与偏好灵活性的统一框架。$λ$-DPO并非将多维偏好信号压缩为单一排序，而是构建一个由概率单纯形上的偏好向量$λ$加权的列表式偏好分布混合模型，从而使单一模型能够内化连续的偏好权衡谱系。为进一步提升鲁棒性，我们引入一种性能驱动的随机$λ$调度器，该调度器基于下游实证性能自适应地采样偏好权重，显式地缓解了静态加权方案固有的设定错误风险。我们在六个广泛使用的基准测试上，跨多种模型家族与规模评估了所提方法。实验结果表明，该方法相较于基线模型取得了持续一致的性能提升。