Recent research has shown the potential of Nash Learning via Human Feedback for large language model alignment by incorporating the notion of a preference model in a minimax game setup. We take this idea further by casting the alignment as a mirror descent algorithm against the adaptive feedback of an improved opponent, thereby removing the need for learning a preference model or the existence of an annotated dataset altogether. The resulting algorithm, which we refer to as Language Alignment via Nash-learning and Adaptive feedback (LANA), is capable of self-alignment without the need for a human-annotated preference dataset. We support this statement with various experiments and mathematical discussion.

翻译：近期研究表明，通过将偏好模型的概念融入极小极大博弈框架，基于人类反馈的纳什学习在大语言模型对齐中展现出巨大潜力。我们进一步发展了这一思想，将对齐问题构建为针对改进对手自适应反馈的镜像下降算法，从而完全消除了学习偏好模型或依赖标注数据集的需求。由此产生的算法——我们称之为基于纳什学习与自适应反馈的语言对齐（LANA）——能够在无需人工标注偏好数据集的情况下实现自我对齐。我们通过多组实验与数学论证支持这一结论。