Offline reinforcement learning (RL) is a challenging setting where existing off-policy actor-critic methods perform poorly due to the overestimation of out-of-distribution state-action pairs. Thus, various additional augmentations are proposed to keep the learned policy close to the offline dataset (or the behavior policy). In this work, starting from the analysis of offline monotonic policy improvement, we get a surprising finding that some online on-policy algorithms are naturally able to solve offline RL. Specifically, the inherent conservatism of these on-policy algorithms is exactly what the offline RL method needs to overcome the overestimation. Based on this, we propose Behavior Proximal Policy Optimization (BPPO), which solves offline RL without any extra constraint or regularization introduced compared to PPO. Extensive experiments on the D4RL benchmark indicate this extremely succinct method outperforms state-of-the-art offline RL algorithms. Our implementation is available at https://github.com/Dragon-Zhuang/BPPO.

翻译：离线强化学习（RL）是一个具有挑战性的场景，现有的离策略演员-评论家方法由于对分布外状态-动作对的高估而表现不佳。为此，研究者提出了各种额外增强方法，以使学习到的策略接近离线数据集（或行为策略）。本文从离线单调策略改进的分析出发，发现了一个令人惊讶的结果：某些在线同策略算法天然能够解决离线RL问题。具体而言，这些同策略算法固有的保守性正是离线RL方法克服高估问题所需的关键因素。基于此，我们提出了行为近端策略优化（BPPO），该方法在PPO基础上未引入任何额外约束或正则化即可解决离线RL问题。在D4RL基准上的大量实验表明，这种极其简洁的方法优于当前最先进的离线RL算法。我们的实现代码已开源至https://github.com/Dragon-Zhuang/BPPO。