Conventional reinforcement learning (RL) needs an environment to collect fresh data, which is impractical when online interactions are costly. Offline RL provides an alternative solution by directly learning from the previously collected dataset. However, it will yield unsatisfactory performance if the quality of the offline datasets is poor. In this paper, we consider an offline-to-online setting where the agent is first learned from the offline dataset and then trained online, and propose a framework called Adaptive Policy Learning for effectively taking advantage of offline and online data. Specifically, we explicitly consider the difference between the online and offline data and apply an adaptive update scheme accordingly, that is, a pessimistic update strategy for the offline dataset and an optimistic/greedy update scheme for the online dataset. Such a simple and effective method provides a way to mix the offline and online RL and achieve the best of both worlds. We further provide two detailed algorithms for implementing the framework through embedding value or policy-based RL algorithms into it. Finally, we conduct extensive experiments on popular continuous control tasks, and results show that our algorithm can learn the expert policy with high sample efficiency even when the quality of offline dataset is poor, e.g., random dataset.

翻译：传统强化学习需要与环境交互来收集新数据，但在在线交互成本高昂时，这种做法并不实用。离线强化学习通过直接利用先前收集的数据集进行学习，提供了一种替代方案。然而，若离线数据集质量较差，该方法会产生不令人满意的效果。本文考虑一种"离线到在线"设置——智能体首先通过离线数据集学习，随后进行在线训练，并提出一种名为自适应策略学习的框架，以有效利用离线与在线数据。具体而言，我们显式考虑在线与离线数据的差异，并据此应用自适应更新策略：对离线数据集采用悲观更新策略，对在线数据集采用乐观/贪婪更新策略。这种简单有效的方法提供了融合离线与在线强化学习、实现二者优势互补的途径。我们进一步提出两种具体算法，通过将基于价值或基于策略的强化学习算法嵌入该框架来实现。最后，我们在流行的连续控制任务上进行了大量实验，结果表明：即使离线数据集质量较差（例如随机数据集），我们的算法仍能以高样本效率学习专家策略。