Ensuring alignment of language models' outputs with human preferences is critical to guarantee a useful, safe, and pleasant user experience. Thus, human alignment has been extensively studied recently and several methods such as Reinforcement Learning from Human Feedback (RLHF), Direct Policy Optimisation (DPO) and Sequence Likelihood Calibration (SLiC) have emerged. In this paper, our contribution is two-fold. First, we show the equivalence between two recent alignment methods, namely Identity Policy Optimisation (IPO) and Nash Mirror Descent (Nash-MD). Second, we introduce a generalisation of IPO, named IPO-MD, that leverages the regularised sampling approach proposed by Nash-MD. This equivalence may seem surprising at first sight, since IPO is an offline method whereas Nash-MD is an online method using a preference model. However, this equivalence can be proven when we consider the online version of IPO, that is when both generations are sampled by the online policy and annotated by a trained preference model. Optimising the IPO loss with such a stream of data becomes then equivalent to finding the Nash equilibrium of the preference model through self-play. Building on this equivalence, we introduce the IPO-MD algorithm that generates data with a mixture policy (between the online and reference policy) similarly as the general Nash-MD algorithm. We compare online-IPO and IPO-MD to different online versions of existing losses on preference data such as DPO and SLiC on a summarisation task.

翻译：确保语言模型输出与人类偏好对齐对于保证有用、安全且令人愉悦的用户体验至关重要。因此，人类对齐近期得到了广泛研究，涌现出多种方法，如基于人类反馈的强化学习（RLHF）、直接策略优化（DPO）和序列似然校准（SLiC）。本文的贡献体现在两方面：首先，我们揭示了两种近期对齐方法——身份策略优化（IPO）和纳什镜像下降（Nash-MD）之间的等价性；其次，我们提出了IPO的泛化版本IPO-MD，该算法采用Nash-MD提出的正则化采样方法。这种等价性初看可能令人惊讶，因为IPO是一种离线方法，而Nash-MD则是使用偏好模型的在线方法。然而，当考虑IPO的在线版本（即生成结果由在线策略采样并由训练好的偏好模型标注）时，这种等价性可以被证明。在此数据流上优化IPO损失等价于通过自博弈寻找偏好模型的纳什均衡。基于此等价性，我们引入IPO-MD算法，该算法与通用Nash-MD算法类似，采用混合策略（在线策略与参考策略的混合）生成数据。我们在摘要任务上，将在线IPO和IPO-MD与DPO、SLiC等现有损失函数的在线版本进行了比较。