Bootstrapping and rollout are two fundamental principles for value function estimation in reinforcement learning (RL). We introduce a novel class of Bellman operators, called subgraph Bellman operators, that interpolate between bootstrapping and rollout methods. Our estimator, derived by solving the fixed point of the empirical subgraph Bellman operator, combines the strengths of the bootstrapping-based temporal difference (TD) estimator and the rollout-based Monte Carlo (MC) methods. Specifically, the error upper bound of our estimator approaches the optimal variance achieved by TD, with an additional term depending on the exit probability of a selected subset of the state space. At the same time, the estimator exhibits the finite-sample adaptivity of MC, with sample complexity depending only on the occupancy measure of this subset. We complement the upper bound with an information-theoretic lower bound, showing that the additional term is unavoidable given a reasonable sample size. Together, these results establish subgraph Bellman estimators as an optimal and adaptive framework for reconciling TD and MC methods in policy evaluation.

翻译：引导和展开是强化学习中价值函数估计的两个基本原则。我们引入了一类新颖的贝尔曼算子，称为子图贝尔曼算子，它在引导和展开方法之间进行插值。通过求解经验子图贝尔曼算子的不动点，我们推导出的估计器结合了基于引导的时间差分估计器和基于展开的蒙特卡洛方法的优点。具体而言，我们估计器的误差上界趋近于TD达到的最优方差，并附加一个取决于状态空间选定子集退出概率的项。同时，该估计器展现出MC的有限样本自适应性，其样本复杂度仅取决于该子集的占用测度。我们用一个信息论下界补充了上界，表明在合理的样本量下，附加项是不可避免的。这些结果共同确立了子图贝尔曼估计器作为一个最优且自适应的框架，用于在策略评估中协调TD和MC方法。