Approximate value iteration (AVI) is a family of algorithms for reinforcement learning (RL) that aims to obtain an approximation of the optimal value function. Generally, AVI algorithms implement an iterated procedure where each step consists of (i) an application of the Bellman operator and (ii) a projection step into a considered function space. Notoriously, the Bellman operator leverages transition samples, which strongly determine its behavior, as uninformative samples can result in negligible updates or long detours, whose detrimental effects are further exacerbated by the computationally intensive projection step. To address these issues, we propose a novel alternative approach based on learning an approximate version of the Bellman operator rather than estimating it through samples as in AVI approaches. This way, we are able to (i) generalize across transition samples and (ii) avoid the computationally intensive projection step. For this reason, we call our novel operator projected Bellman operator (PBO). We formulate an optimization problem to learn PBO for generic sequential decision-making problems, and we theoretically analyze its properties in two representative classes of RL problems. Furthermore, we theoretically study our approach under the lens of AVI and devise algorithmic implementations to learn PBO in offline and online settings by leveraging neural network parameterizations. Finally, we empirically showcase the benefits of PBO w.r.t. the regular Bellman operator on several RL problems.

翻译：近似值迭代（AVI）是一类用于强化学习（RL）的算法，旨在获得最优值函数的近似。通常，AVI算法实现一个迭代过程，每一步包括：(i) 应用贝尔曼算子，(ii) 将结果投影到考虑的函数空间。众所周知，贝尔曼算子利用转移样本，这些样本强烈决定了其行为，因为信息量不足的样本可能导致微不足道的更新或长时间绕路，而计算密集的投影步骤进一步加剧了这种不利影响。为了解决这些问题，我们提出了一种新颖的替代方法，该方法基于学习贝尔曼算子的近似版本，而不是像AVI方法那样通过样本估算它。这样，我们能够(i) 在转移样本间进行泛化，以及(ii) 避免计算密集的投影步骤。因此，我们将这种新颖算子称为投影贝尔曼算子（PBO）。我们为通用序列决策问题制定了一个优化问题来学习PBO，并在两类代表性RL问题中从理论上分析了其性质。此外，我们从AVI的角度对我们的方法进行了理论研究，并设计了利用神经网络参数化的算法实现，以在离线和在线设置中学习PBO。最后，我们在多个RL问题上实证展示了PBO相对于常规贝尔曼算子的优势。