Approximate value iteration (AVI) is a family of algorithms for reinforcement learning (RL) that aims to obtain an approximation of the optimal value function. Generally, AVI algorithms implement an iterated procedure where each step consists of (i) an application of the Bellman operator and (ii) a projection step into a considered function space. Notoriously, the Bellman operator leverages transition samples, which strongly determine its behavior, as uninformative samples can result in negligible updates or long detours, whose detrimental effects are further exacerbated by the computationally intensive projection step. To address these issues, we propose a novel alternative approach based on learning an approximate version of the Bellman operator rather than estimating it through samples as in AVI approaches. This way, we are able to (i) generalize across transition samples and (ii) avoid the computationally intensive projection step. For this reason, we call our novel operator projected Bellman operator (PBO). We formulate an optimization problem to learn PBO for generic sequential decision-making problems, and we theoretically analyze its properties in two representative classes of RL problems. Furthermore, we theoretically study our approach under the lens of AVI and devise algorithmic implementations to learn PBO in offline and online settings by leveraging neural network parameterizations. Finally, we empirically showcase the benefits of PBO w.r.t. the regular Bellman operator on several RL problems.

翻译：近似值迭代（AVI）是一类用于强化学习（RL）的算法，旨在获得最优值函数的近似。通常，AVI算法实现迭代过程，每一步包括：（i）应用贝尔曼算子，（ii）投影到所考虑的函数空间。众所周知，贝尔曼算子利用转移样本，这些样本强烈决定其行为——无信息样本可能导致忽略不计的更新或长时间绕路，而计算密集的投影步骤进一步加剧了这些不利影响。为解决这些问题，我们提出了一种新颖的替代方法，该方法基于学习贝尔曼算子的近似版本，而非像AVI方法那样通过样本估计算子。通过这种方式，我们能够（i）跨转移样本进行泛化，并（ii）避免计算密集的投影步骤。因此，我们将新算子称为投影贝尔曼算子（PBO）。我们为通用序列决策问题制定了学习PBO的优化问题，并在两类代表性RL问题中从理论上分析了其性质。此外，我们从AVI视角对方法进行理论分析，并设计了利用神经网络参数化在离线与在线设置中学习PBO的算法实现。最后，我们通过多个RL问题实证展示了PBO相对于常规贝尔曼算子的优势。