We provide a theoretical framework for Reinforcement Learning with Human Feedback (RLHF). Our analysis shows that when the true reward function is linear, the widely used maximum likelihood estimator (MLE) converges under both the Bradley-Terry-Luce (BTL) model and the Plackett-Luce (PL) model. However, we show that when training a policy based on the learned reward model, MLE fails while a pessimistic MLE provides policies with improved performance under certain coverage assumptions. Additionally, we demonstrate that under the PL model, the true MLE and an alternative MLE that splits the $K$-wise comparison into pairwise comparisons both converge. Moreover, the true MLE is asymptotically more efficient. Our results validate the empirical success of existing RLHF algorithms in InstructGPT and provide new insights for algorithm design. Furthermore, our results unify the problem of RLHF and max-entropy Inverse Reinforcement Learning (IRL), and provide the first sample complexity bound for max-entropy IRL.

翻译：我们为带人类反馈的强化学习（RLHF）提供了理论框架。分析表明，当真实奖励函数为线性时，广泛使用的极大似然估计（MLE）在Bradley-Terry-Luce（BTL）模型和Plackett-Luce（PL）模型下均收敛。然而，我们发现在基于学习到的奖励模型训练策略时，MLE方法失效，而悲观MLE在特定覆盖假设下能提供性能更优的策略。此外，我们证明了在PL模型下，真实MLE与将$K$元比较拆分为成对比较的替代MLE均收敛，且真实MLE在渐近意义上更高效。我们的结果验证了InstructGPT中现有RLHF算法的实证成功，并为算法设计提供了新见解。进一步地，我们的研究统一了RLHF与最大熵逆强化学习（IRL）问题，并首次给出了最大熵IRL的样本复杂度界。