Aligning large language models (LLMs) with human preferences has proven effective for enhancing model capabilities, yet standard preference modeling using the Bradley-Terry model assumes transitivity, overlooking the inherent complexity of human population preferences. Nash learning from human feedback (NLHF) addresses this by framing non-transitive preferences as a two-player zero-sum game, where alignment reduces to finding the Nash equilibrium (NE). However, existing algorithms typically rely on regularization, incurring unavoidable bias when computing the duality gap in the original game. In this work, we provide the first convergence guarantee for Optimistic Multiplicative Weights Update ($\mathtt{OMWU}$) in NLHF, showing that it achieves last-iterate linear convergence after a burn-in phase whenever an NE with full support exists, with an instance-dependent linear convergence rate to the original NE, measured by duality gaps. Compared to prior results in Wei et al. (2020), we do not require the assumption of NE uniqueness. Our analysis identifies a novel marginal convergence behavior, where the probability of rarely played actions grows exponentially from exponentially small values, enabling exponentially better dependence on instance-dependent constants than prior results. Experiments corroborate the theoretical strengths of $\mathtt{OMWU}$ in both tabular and neural policy classes, demonstrating its potential for LLM applications.

翻译：将大型语言模型（LLM）与人类偏好对齐已被证明能有效提升模型能力，然而基于Bradley-Terry模型的标准偏好建模假设了传递性，忽略了人类群体偏好固有的复杂性。基于人类反馈的纳什学习（NLHF）通过将非传递性偏好建模为两人零和博弈来解决此问题，其中对齐问题可简化为寻找纳什均衡（NE）。然而，现有算法通常依赖正则化，在计算原始博弈的对偶间隙时会产生不可避免的偏差。本研究中，我们首次为NLHF框架下的乐观乘性权重更新算法（$\mathtt{OMWU}$）提供了收敛性保证，证明当存在具有完全支撑的纳什均衡时，该算法在预热阶段后能实现末次迭代线性收敛，并以对偶间隙度量的实例相关线性收敛速率收敛至原始纳什均衡。相较于Wei等人（2020）的先前结果，我们无需纳什均衡唯一性的假设。我们的分析揭示了一种新颖的边缘收敛行为：极少被采用动作的概率会从指数级小的初始值呈指数增长，这使得算法对实例相关常数的依赖度较先前结果呈指数级改善。实验在表格策略与神经网络策略两类设置中均验证了$\mathtt{OMWU}$的理论优势，展现了其在LLM应用中的潜力。