A recently popular approach to solving reinforcement learning is with data from human preferences. In fact, human preference data are now used with classic reinforcement learning algorithms such as actor-critic methods, which involve evaluating an intermediate policy over a reward learned from human preference data with distribution shift, known as off-policy evaluation (OPE). Such algorithm includes (i) learning reward function from human preference dataset, and (ii) learning expected cumulative reward of a target policy. Despite the huge empirical success, existing OPE methods with preference data often lack theoretical understanding and rely heavily on heuristics. In this paper, we study the sample efficiency of OPE with human preference and establish a statistical guarantee for it. Specifically, we approach OPE by learning the value function by fitted-Q-evaluation with a deep neural network. By appropriately selecting the size of a ReLU network, we show that one can leverage any low-dimensional manifold structure in the Markov decision process and obtain a sample-efficient estimator without suffering from the curse of high data ambient dimensionality. Under the assumption of high reward smoothness, our results \textit{almost align with the classical OPE results with observable reward data}. To the best of our knowledge, this is the first result that establishes a \textit{provably efficient} guarantee for off-policy evaluation with RLHF.

翻译：一种近期流行的强化学习解决方案是利用人类偏好数据。事实上，人类偏好数据现已与经典强化学习算法（如行为者-评论家方法）结合使用，这类算法需在由人类偏好数据学习得到的奖励函数上评估中间策略，并面临分布偏移问题，即离线策略评估（OPE）。此类算法包括：（i）从人类偏好数据集中学习奖励函数，（ii）学习目标策略的期望累积奖励。尽管实证上取得了巨大成功，现有的基于偏好数据的OPE方法往往缺乏理论理解，且严重依赖启发式方法。本文研究了基于人类偏好的OPE的样本效率问题，并建立了其统计保证。具体而言，我们通过使用深度神经网络进行拟合Q评估来学习价值函数，从而解决OPE问题。通过适当选择ReLU网络的规模，我们证明可以利用马尔可夫决策过程中的任意低维流形结构，获得一个样本高效的估计器，而无需受限于高维数据环境维度的灾难。在奖励高光滑性的假设下，我们的结果几乎与使用可观测奖励数据的经典OPE结果一致。据我们所知，这是首次为基于人类反馈的强化学习（RLHF）的离线策略评估建立可证明高效的保证。