A recently popular approach to solving reinforcement learning is with data from human preferences. In fact, human preference data are now used with classic reinforcement learning algorithms such as actor-critic methods, which involve evaluating an intermediate policy over a reward learned from human preference data with distribution shift, known as off-policy evaluation (OPE). Such algorithm includes (i) learning reward function from human preference dataset, and (ii) learning expected cumulative reward of a target policy. Despite the huge empirical success, existing OPE methods with preference data often lack theoretical understanding and rely heavily on heuristics. In this paper, we study the sample efficiency of OPE with human preference and establish a statistical guarantee for it. Specifically, we approach OPE by learning the value function by fitted-Q-evaluation with a deep neural network. By appropriately selecting the size of a ReLU network, we show that one can leverage any low-dimensional manifold structure in the Markov decision process and obtain a sample-efficient estimator without suffering from the curse of high data ambient dimensionality. Under the assumption of high reward smoothness, our results \textit{almost align with the classical OPE results with observable reward data}. To the best of our knowledge, this is the first result that establishes a \textit{provably efficient} guarantee for off-policy evaluation with RLHF.

翻译：一种近期流行的强化学习方法利用人类偏好数据。事实上，人类偏好数据现已被应用于经典强化学习算法（如演员-评论家方法），这些方法涉及在从人类偏好数据中学习的奖励函数上评估中间策略，同时面临分布偏移问题，即离线策略评估（OPE）。此类算法包括：（i）从人类偏好数据集中学习奖励函数，以及（ii）学习目标策略的期望累积奖励。尽管取得了巨大的实证成功，现有基于偏好数据的OPE方法通常缺乏理论基础，并严重依赖启发式方法。本文研究了基于人类偏好的OPE的样本效率，并为其建立了统计保证。具体而言，我们通过使用深度神经网络进行拟合Q值评估来学习价值函数，从而解决OPE问题。通过适当选择ReLU网络的大小，我们证明可以利用马尔可夫决策过程中的任意低维流形结构，获得一个样本高效的估计器，而无需遭受高数据环境维度的诅咒。在奖励高度平滑的假设下，我们的结果几乎与经典的有观测奖励数据的OPE结果一致。据我们所知，这是首个为RLHF中的离线策略评估建立可证明高效保证的结果。