Reinforcement Learning algorithms that learn from human feedback (RLHF) need to be efficient in terms of statistical complexity, computational complexity, and query complexity. In this work, we consider the RLHF setting where the feedback is given in the format of preferences over pairs of trajectories. In the linear MDP model, by using randomization in algorithm design, we present an algorithm that is sample efficient (i.e., has near-optimal worst-case regret bounds) and has polynomial running time (i.e., computational complexity is polynomial with respect to relevant parameters). Our algorithm further minimizes the query complexity through a novel randomized active learning procedure. In particular, our algorithm demonstrates a near-optimal tradeoff between the regret bound and the query complexity. To extend the results to more general nonlinear function approximation, we design a model-based randomized algorithm inspired by the idea of Thompson sampling. Our algorithm minimizes Bayesian regret bound and query complexity, again achieving a near-optimal tradeoff between these two quantities. Computation-wise, similar to the prior Thompson sampling algorithms under the regular RL setting, the main computation primitives of our algorithm are Bayesian supervised learning oracles which have been heavily investigated on the empirical side when applying Thompson sampling algorithms to RL benchmark problems.

翻译：基于人类反馈的强化学习（RLHF）算法需在统计复杂度、计算复杂度和查询复杂度三方面实现高效。本研究考虑了以轨迹对偏好形式提供反馈的RLHF场景。在线性马尔可夫决策过程模型框架下，我们通过引入算法设计的随机化思想，提出了一种具有样本高效性（即实现近似最优的最坏情况遗憾界）且运行时间多项式（即计算复杂度关于相关参数呈多项式级）的算法。该算法通过新颖的随机化主动学习流程进一步最小化查询复杂度：具体而言，我们的算法在遗憾界与查询复杂度之间实现了近似最优的权衡。为将结果推广至更广泛的非线性函数逼近场景，我们受汤普森采样思想启发设计了一种基于模型的随机化算法。该算法在最小化贝叶斯遗憾界与查询复杂度的同时，再次实现了两者的近似最优权衡。计算层面，与标准强化学习场景中的现有汤普森采样算法类似，本算法的主要计算基元是贝叶斯监督学习预言机——该类预言机在汤普森采样算法应用于强化学习基准问题的实证研究中已得到广泛探究。