Offline RL methods have been shown to reduce the need for environment interaction by training agents using offline collected episodes. However, these methods typically require action information to be logged during data collection, which can be difficult or even impossible in some practical cases. In this paper, we investigate the potential of using action-free offline datasets to improve online reinforcement learning, name this problem Reinforcement Learning with Action-Free Offline Pretraining (AFP-RL). We introduce Action-Free Guide (AF-Guide), a method that guides online training by extracting knowledge from action-free offline datasets. AF-Guide consists of an Action-Free Decision Transformer (AFDT) implementing a variant of Upside-Down Reinforcement Learning. It learns to plan the next states from the offline dataset, and a Guided Soft Actor-Critic (Guided SAC) that learns online with guidance from AFDT. Experimental results show that AF-Guide can improve sample efficiency and performance in online training thanks to the knowledge from the action-free offline dataset. Code is available at https://github.com/Vision-CAIR/AF-Guide.

翻译：离线RL方法通过使用离线收集的回合进行训练，已被证明能减少与环境交互的需求。然而，这些方法通常需要在数据采集期间记录动作信息，这在某些实际场景中可能困难甚至无法实现。本文探究了利用无动作离线数据集改进在线强化学习的潜力，将此问题命名为无动作离线预训练强化学习（AFP-RL）。我们提出无动作引导（AF-Guide）方法，通过从无动作离线数据集中提取知识来引导在线训练。AF-Guide由两部分组成：实现倒置强化学习变体的无动作决策Transformer（AFDT），它学习从离线数据集中规划后续状态；以及引导型软演员-评论家算法（Guided SAC），它在AFDT的引导下进行在线学习。实验结果表明，AF-Guide借助无动作离线数据集中的知识，能提升在线训练的样本效率和性能。代码开源在https://github.com/Vision-CAIR/AF-Guide。