Offline RL methods have been shown to reduce the need for environment interaction by training agents using offline collected episodes. However, these methods typically require action information to be logged during data collection, which can be difficult or even impossible in some practical cases. In this paper, we investigate the potential of using action-free offline datasets to improve online reinforcement learning, name this problem Reinforcement Learning with Action-Free Offline Pretraining (AFP-RL). We introduce Action-Free Guide (AF-Guide), a method that guides online training by extracting knowledge from action-free offline datasets. AF-Guide consists of an Action-Free Decision Transformer (AFDT) implementing a variant of Upside-Down Reinforcement Learning. It learns to plan the next states from the offline dataset, and a Guided Soft Actor-Critic (Guided SAC) that learns online with guidance from AFDT. Experimental results show that AF-Guide can improve sample efficiency and performance in online training thanks to the knowledge from the action-free offline dataset.

翻译：离线强化学习方法通过利用离线收集的经验片段训练智能体，已被证明能减少对与环境交互的需求。然而，这些方法通常需要在数据收集期间记录动作信息，这在某些实际场景中可能难以实现甚至完全无法完成。本文探究了利用无动作离线数据集改进在线强化学习的潜力，并将该问题命名为“无动作离线预训练强化学习”（AFP-RL）。我们提出无动作引导器（AF-Guide）方法，该方法通过从无动作离线数据集中提取知识来指导在线训练。AF-Guide包含两个核心组件：实现倒置强化学习变体的无动作决策Transformer（AFDT），用于从离线数据集中学习规划后续状态；以及受AFDT指导的在线学习算法引导型软演员-评论家（Guided SAC）。实验结果表明，得益于无动作离线数据集中的知识，AF-Guide能够提升在线训练的样本效率与性能表现。