In this paper, we investigate the problem of offline Preference-based Reinforcement Learning (PbRL) with human feedback where feedback is available in the form of preference between trajectory pairs rather than explicit rewards. Our proposed algorithm consists of two main steps: (1) estimate the implicit reward using Maximum Likelihood Estimation (MLE) with general function approximation from offline data and (2) solve a distributionally robust planning problem over a confidence set around the MLE. We consider the general reward setting where the reward can be defined over the whole trajectory and provide a novel guarantee that allows us to learn any target policy with a polynomial number of samples, as long as the target policy is covered by the offline data. This guarantee is the first of its kind with general function approximation. To measure the coverage of the target policy, we introduce a new single-policy concentrability coefficient, which can be upper bounded by the per-trajectory concentrability coefficient. We also establish lower bounds that highlight the necessity of such concentrability and the difference from standard RL, where state-action-wise rewards are directly observed. We further extend and analyze our algorithm when the feedback is given over action pairs.

翻译：本文研究了基于人类反馈的离线偏好基础强化学习（PbRL）问题，其中反馈以轨迹对之间的偏好形式提供，而非显式奖励。我们提出的算法包含两大步骤：（1）利用离线数据通过通用函数逼近的最大似然估计（MLE）估计隐式奖励；（2）在MLE周围置信集上求解分布鲁棒规划问题。我们考虑奖励可定义在整个轨迹上的通用奖励设置，并提供一种新颖的保证：只要目标策略被离线数据覆盖，即可通过多项式数量的样本学习任意目标策略。该保证是首个在通用函数逼近框架下实现的此类结果。为衡量目标策略的覆盖度，我们引入新的单策略集中性系数，其可由每条轨迹的集中性系数上界界定。我们还建立了下界，凸显此类集中性的必要性以及与直接观测状态-动作级奖励的标准强化学习之间的差异。进一步地，我们扩展并分析了反馈基于动作对时的算法。