While originally developed for continuous control problems, Proximal Policy Optimization (PPO) has emerged as the work-horse of a variety of reinforcement learning (RL) applications, including the fine-tuning of generative models. Unfortunately, PPO requires multiple heuristics to enable stable convergence (e.g. value networks, clipping), and is notorious for its sensitivity to the precise implementation of these components. In response, we take a step back and ask what a minimalist RL algorithm for the era of generative models would look like. We propose REBEL, an algorithm that cleanly reduces the problem of policy optimization to regressing the relative reward between two completions to a prompt in terms of the policy, enabling strikingly lightweight implementation. In theory, we prove that fundamental RL algorithms like Natural Policy Gradient can be seen as variants of REBEL, which allows us to match the strongest known theoretical guarantees in terms of convergence and sample complexity in the RL literature. REBEL can also cleanly incorporate offline data and be extended to handle the intransitive preferences we frequently see in practice. Empirically, we find that REBEL provides a unified approach to language modeling and image generation with stronger or similar performance as PPO and DPO, all while being simpler to implement and more computationally efficient than PPO. When fine-tuning Llama-3-8B-Instruct, REBEL achieves strong performance in AlpacaEval 2.0, MT-Bench, and Open LLM Leaderboard.

翻译：尽管近端策略优化（PPO）最初是为连续控制问题开发的，但它已成为各类强化学习（RL）应用（包括生成模型微调）的核心算法。然而，PPO需要多种启发式方法（如价值网络、裁剪）才能实现稳定收敛，并且因其对这些组件实现细节的敏感性而众所周知。为此，我们退后一步思考：在生成模型时代，一个极简的强化学习算法应是什么样。我们提出了REBEL，该算法清晰地将策略优化问题转化为根据策略回归提示词两个补全结果之间的相对奖励，从而实现极其轻量级的实现。理论上，我们证明了自然策略梯度等基础RL算法可视为REBEL的变体，这使我们能够匹配RL文献中在收敛性和样本复杂度方面已知的最强理论保证。REBEL还能无缝整合离线数据，并可扩展以处理实践中常见的非传递性偏好。实证表明，REBEL为语言建模和图像生成提供了统一方法，其性能优于或相当于PPO和DPO，同时比PPO更易于实现且计算效率更高。在对Llama-3-8B-Instruct进行微调时，REBEL在AlpacaEval 2.0、MT-Bench和Open LLM Leaderboard中均取得了优异表现。