The core challenge of offline reinforcement learning (RL) is dealing with the (potentially catastrophic) extrapolation error induced by the distribution shift between the history dataset and the desired policy. A large portion of prior work tackles this challenge by implicitly/explicitly regularizing the learning policy towards the behavior policy, which is hard to estimate reliably in practice. In this work, we propose to regularize towards the Q-function of the behavior policy instead of the behavior policy itself, under the premise that the Q-function can be estimated more reliably and easily by a SARSA-style estimate and handles the extrapolation error more straightforwardly. We propose two algorithms taking advantage of the estimated Q-function through regularizations, and demonstrate they exhibit strong performance on the D4RL benchmarks.

翻译：离线强化学习的核心挑战在于处理由历史数据集与目标策略之间分布偏移引发的（可能灾难性的）外推误差。大量现有工作通过隐式或显式地使学习策略向行为策略（behavior policy）正则化来解决这一挑战，然而在实践中行为策略难以被可靠估计。本文提出替代方案：基于行为策略的Q函数（而非行为策略本身）进行正则化。其前提是Q函数可通过SARSA类方法更可靠、更简便地估计，并能更直接地处理外推误差。我们设计了两种利用正则化过程中估计的Q函数的算法，并在D4RL基准测试中验证了其优越性能。