Reinforcement Learning from Human Feedback (RLHF) has emerged as a popular paradigm for aligning models with human intent. Typically RLHF algorithms operate in two phases: first, use human preferences to learn a reward function and second, align the model by optimizing the learned reward via reinforcement learning (RL). This paradigm assumes that human preferences are distributed according to reward, but recent work suggests that they instead follow the regret under the user's optimal policy. Thus, learning a reward function from feedback is not only based on a flawed assumption of human preference, but also leads to unwieldy optimization challenges that stem from policy gradients or bootstrapping in the RL phase. Because of these optimization challenges, contemporary RLHF methods restrict themselves to contextual bandit settings (e.g., as in large language models) or limit observation dimensionality (e.g., state-based robotics). We overcome these limitations by introducing a new family of algorithms for optimizing behavior from human feedback using the regret-based model of human preferences. Using the principle of maximum entropy, we derive Contrastive Preference Learning (CPL), an algorithm for learning optimal policies from preferences without learning reward functions, circumventing the need for RL. CPL is fully off-policy, uses only a simple contrastive objective, and can be applied to arbitrary MDPs. This enables CPL to elegantly scale to high-dimensional and sequential RLHF problems while being simpler than prior methods.

翻译：基于人类反馈的强化学习（RLHF）已成为将模型与人类意图对齐的流行范式。典型的RLHF算法分为两个阶段：首先利用人类偏好学习奖励函数，其次通过强化学习（RL）优化所学奖励来对齐模型。这一范式假设人类偏好服从奖励分布，但近期研究表明偏好实际上遵循用户最优策略下的遗憾值。因此，从反馈中学习奖励函数不仅基于对人类偏好的错误假设，还会导致RL阶段策略梯度或自举法带来的复杂优化挑战。由于这些优化困难，当代RLHF方法将其应用限制在上下文赌博机环境（如大语言模型）或限制观测维度（如基于状态的机器人技术）。我们通过引入基于人类偏好遗憾模型的新算法族克服了这些局限。利用最大熵原理，我们推导出对比偏好学习（CPL）——一种无需学习奖励函数即可从偏好中学习最优策略的算法，规避了RL的需求。CPL完全采用离线策略，仅需简单的对比目标函数，可应用于任意马尔可夫决策过程（MDP）。这使得CPL能够优雅地扩展到高维和序列化RLHF问题，同时比现有方法更为简洁。