Reinforcement Learning algorithms that learn from human feedback (RLHF) need to be efficient in terms of statistical complexity, computational complexity, and query complexity. In this work, we consider the RLHF setting where the feedback is given in the format of preferences over pairs of trajectories. In the linear MDP model, using randomization in algorithm design, we present an algorithm that is sample efficient (i.e., has near-optimal worst-case regret bounds) and has polynomial running time (i.e., computational complexity is polynomial with respect to relevant parameters). Our algorithm further minimizes the query complexity through a novel randomized active learning procedure. In particular, our algorithm demonstrates a near-optimal tradeoff between the regret bound and the query complexity. To extend the results to more general nonlinear function approximation, we design a model-based randomized algorithm inspired by the idea of Thompson sampling. Our algorithm minimizes Bayesian regret bound and query complexity, again achieving a near-optimal tradeoff between these two quantities. Computation-wise, similar to the prior Thompson sampling algorithms under the regular RL setting, the main computation primitives of our algorithm are Bayesian supervised learning oracles which have been heavily investigated on the empirical side when applying Thompson sampling algorithms to RL benchmark problems.

翻译：从人类反馈中学习的强化学习算法(RLHF)需要在统计复杂度、计算复杂度和查询复杂度方面具有高效性。本研究考虑反馈以轨迹对偏好形式给出的RLHF场景。在线性MDP模型中，通过算法设计中的随机化技术，我们提出了一种兼具样本效率（即具有接近最优的最坏情况遗憾界）与多项式运行时间（即计算复杂度相对于相关参数呈多项式级）的算法。该算法通过新颖的随机化主动学习过程进一步最小化查询复杂度，具体而言，其在遗憾界与查询复杂度之间实现了接近最优的权衡。为将结果推广至更一般的非线性函数逼近场景，我们受汤普森采样思想启发设计了基于模型的随机化算法。该算法同时最小化贝叶斯遗憾界与查询复杂度，再次实现两者间的接近最优权衡。在计算层面，与常规RL场景下的现有汤普森采样算法类似，本算法的主要计算原语为贝叶斯监督学习预言机——该技术已在将汤普森采样算法应用于RL基准问题的实证研究中得到充分验证。