Direct alignment from preferences (DAP) methods, such as DPO, have recently emerged as efficient alternatives to reinforcement learning from human feedback (RLHF), that do not require a separate reward model. However, the preference datasets used in DAP methods are usually collected ahead of training and never updated, thus the feedback is purely offline. Moreover, responses in these datasets are often sampled from a language model distinct from the one being aligned, and since the model evolves over training, the alignment phase is inevitably off-policy. In this study, we posit that online feedback is key and improves DAP methods. Our method, online AI feedback (OAIF), uses an LLM as annotator: on each training iteration, we sample two responses from the current model and prompt the LLM annotator to choose which one is preferred, thus providing online feedback. Despite its simplicity, we demonstrate via human evaluation in several tasks that OAIF outperforms both offline DAP and RLHF methods. We further show that the feedback leveraged in OAIF is easily controllable, via instruction prompts to the LLM annotator.

翻译：从偏好中直接对齐（DAP）方法，如DPO，最近已成为从人类反馈中进行强化学习（RLHF）的高效替代方案，且无需单独的奖励模型。然而，DAP方法中使用的偏好数据集通常是在训练前收集并从未更新，因此反馈完全是离线的。此外，这些数据集中的响应通常采自与待对齐模型不同的语言模型，并且由于模型在训练过程中不断演变，对齐阶段不可避免地存在离策略问题。在本研究中，我们认为在线反馈是关键因素，能够改进DAP方法。我们的方法——在线AI反馈（OAIF）——利用大型语言模型作为标注器：在每次训练迭代中，我们从当前模型中采样两个响应，并提示LLM标注器选择更优的一个，从而提供在线反馈。尽管方法简单，但通过多项任务的人工评估，我们证明OAIF优于离线DAP和RLHF方法。我们进一步表明，通过向LLM标注器提供指令提示，OAIF中所利用的反馈易于控制。