Approximate value iteration~(AVI) is a family of algorithms for reinforcement learning~(RL) that aims to obtain an approximation of the optimal value function. Generally, AVI algorithms implement an iterated procedure where each step consists of (i) an application of the Bellman operator and (ii) a projection step into a considered function space. Notoriously, the Bellman operator leverages transition samples, which strongly determine its behavior, as uninformative samples can result in negligible updates or long detours, whose detrimental effects are further exacerbated by the computationally intensive projection step. To address these issues, we propose a novel alternative approach based on learning an approximate version of the Bellman operator rather than estimating it through samples as in AVI approaches. This way, we are able to (i) generalize across transition samples and (ii) avoid the computationally intensive projection step. For this reason, we call our novel operator projected Bellman operator (PBO). We formulate an optimization problem to learn PBO for generic sequential decision-making problems, and we theoretically analyze its properties in two representative classes of RL problems. Furthermore, we theoretically study our approach under the lens of AVI and devise algorithmic implementations to learn PBO in offline and online settings by leveraging neural network parameterizations. Finally, we empirically showcase the benefits of PBO w.r.t. the regular Bellman operator on several RL problems.

翻译：近似值迭代（AVI）是一类用于强化学习（RL）的算法族，旨在获得最优值函数的近似解。通常，AVI算法通过迭代过程实现，每一步包含：（i）应用贝尔曼算子；（ii）将结果投影到特定函数空间。值得注意的是，贝尔曼算子依赖转移样本，其行为很大程度上受样本质量影响——无信息样本可能导致更新幅度极小或训练路径显著延长，而计算密集的投影步骤会进一步加剧这些负面影响。为解决上述问题，我们提出一种新型替代方法：通过学习贝尔曼算子的近似版本，而非像AVI方法那样通过采样进行估计。通过这种方式，我们能够（i）实现跨转移样本的泛化，（ii）避免计算密集的投影步骤。基于此，我们将新算子命名为投影贝尔曼算子（PBO）。我们针对通用序贯决策问题构建了学习PBO的优化框架，并在两类典型RL问题中对其性质进行理论分析。此外，我们从AVI视角开展理论研究，并设计出利用神经网络参数化在离线与在线场景下学习PBO的算法实现。最后，我们通过多个RL问题实验实证展示了PBO相较于标准贝尔曼算子的优势。