We introduce Adversarial Policy Optimization (AdvPO), a novel solution to the pervasive issue of reward over-optimization in Reinforcement Learning from Human Feedback (RLHF) for Large Language Models (LLMs). Over-optimization occurs when a reward model serves as an imperfect proxy for human preference, and RL-driven policy optimization erroneously exploits reward inaccuracies. In this paper, we begin by introducing a lightweight way to quantify uncertainties in rewards, relying solely on the last layer embeddings of the reward model, without the need for computationally expensive reward ensembles. AdvPO then addresses a distributionally robust optimization problem centred around the confidence interval of the reward model's predictions for policy improvement. Through comprehensive experiments on the Anthropic HH and TL;DR summarization datasets, we illustrate the efficacy of AdvPO in mitigating the overoptimization issue, consequently resulting in enhanced performance as evaluated through human-assisted evaluation.

翻译：我们提出了对抗性策略优化（AdvPO），一种针对大型语言模型（LLM）基于人类反馈的强化学习（RLHF）中普遍存在的奖励过优化问题的新颖解决方案。当奖励模型作为人类偏好的不完美代理，且RL驱动的策略优化错误地利用了奖励的不准确性时，就会发生过优化。在本文中，我们首先引入一种轻量级方法来量化奖励的不确定性，该方法仅依赖于奖励模型的最后一层嵌入，无需计算成本高昂的奖励模型集成。AdvPO随后围绕奖励模型预测的置信区间解决一个以策略改进为中心的分布鲁棒优化问题。通过在Anthropic HH和TL;DR摘要数据集上的全面实验，我们阐明了AdvPO在缓解过优化问题方面的有效性，从而通过人工辅助评估实现了性能提升。