Varied approaches for aligning language models have been proposed, including supervised fine-tuning, RLHF, and direct optimization methods such as DPO. Although DPO has rapidly gained popularity due to its straightforward training process and competitive results, there is an open question of whether there remain practical advantages of using a discriminator, like a reward model, to evaluate responses. We propose D2PO, discriminator-guided DPO, an approach for the online setting where preferences are being collected throughout learning. As we collect gold preferences, we use these not only to train our policy, but to train a discriminative response evaluation model to silver-label even more synthetic data for policy training. We explore this approach across a set of diverse tasks, including a realistic chat setting, we find that our approach leads to higher-quality outputs compared to DPO with the same data budget, and greater efficiency in terms of preference data requirements. Furthermore, we show conditions under which silver labeling is most helpful: it is most effective when training the policy with DPO, outperforming traditional PPO, and benefits from maintaining a separate discriminator from the policy model.

翻译：为对齐语言模型，已有多种方法被提出，包括监督微调、RLHF以及直接优化方法（如DPO）。尽管DPO因其简洁的训练流程和具有竞争力的结果而迅速普及，但关于使用判别器（如奖励模型）评估响应是否仍具有实际优势这一问题尚未明确。我们提出D2PO（判别器引导的DPO），一种在在线设置中随学习过程收集偏好的方法。在收集人工偏好时，我们不仅将其用于训练策略，还用于训练一个判别式响应评估模型，从而为策略训练标注更多合成数据。我们在一系列多样化任务（包括真实聊天场景）中探索了该方法，发现与使用相同数据预算的DPO相比，我们的方法能产生更高质量的输出，并在偏好数据需求上具有更高效率。此外，我们揭示了银标签标注最有效的条件：当使用DPO训练策略时效果最佳（优于传统PPO），且优势在于将判别器与策略模型保持分离。