Reinforcement learning from Human Feedback (RLHF) learns from preference signals, while standard Reinforcement Learning (RL) directly learns from reward signals. Preferences arguably contain less information than rewards, which makes preference-based RL seemingly more difficult. This paper theoretically proves that, for a wide range of preference models, we can solve preference-based RL directly using existing algorithms and techniques for reward-based RL, with small or no extra costs. Specifically, (1) for preferences that are drawn from reward-based probabilistic models, we reduce the problem to robust reward-based RL that can tolerate small errors in rewards; (2) for general arbitrary preferences where the objective is to find the von Neumann winner, we reduce the problem to multiagent reward-based RL which finds Nash equilibria for factored Markov games under a restricted set of policies. The latter case can be further reduce to adversarial MDP when preferences only depend on the final state. We instantiate all reward-based RL subroutines by concrete provable algorithms, and apply our theory to a large class of models including tabular MDPs and MDPs with generic function approximation. We further provide guarantees when K-wise comparisons are available.

翻译：基于人类反馈的强化学习（RLHF）从偏好信号中学习，而标准强化学习（RL）直接从奖励信号中学习。偏好信息量通常少于奖励信息，这使得基于偏好的强化学习看似更加困难。本文在理论上证明：对于一大类偏好模型，我们可以直接使用现有基于奖励的强化学习算法与技术来解决基于偏好的强化学习问题，且额外代价极小甚至为零。具体而言：（1）对于基于奖励概率模型生成的偏好，我们将问题简化为能容忍微小奖励误差的鲁棒奖励强化学习；（2）对于目标为寻找冯·诺依曼胜者的一般任意偏好，我们将其简化为基于多智能体奖励强化学习问题——在受限策略集下求解因子化马尔可夫博弈的纳什均衡。当偏好仅取决于最终状态时，后者可进一步简化为对抗式马尔可夫决策过程。我们通过具体可证明算法实例化所有基于奖励的强化学习子程序，并将理论应用于包括表格型MDP和通用函数近似MDP在内的广泛模型类别。此外，我们还给出了在可获取K元比较时的理论保证。