Sample efficiency and exploration remain major challenges in online reinforcement learning (RL). A powerful approach that can be applied to address these issues is the inclusion of offline data, such as prior trajectories from a human expert or a sub-optimal exploration policy. Previous methods have relied on extensive modifications and additional complexity to ensure the effective use of this data. Instead, we ask: can we simply apply existing off-policy methods to leverage offline data when learning online? In this work, we demonstrate that the answer is yes; however, a set of minimal but important changes to existing off-policy RL algorithms are required to achieve reliable performance. We extensively ablate these design choices, demonstrating the key factors that most affect performance, and arrive at a set of recommendations that practitioners can readily apply, whether their data comprise a small number of expert demonstrations or large volumes of sub-optimal trajectories. We see that correct application of these simple recommendations can provide a $\mathbf{2.5\times}$ improvement over existing approaches across a diverse set of competitive benchmarks, with no additional computational overhead. We have released our code at https://github.com/ikostrikov/rlpd.

翻译：样本效率与探索仍是在线强化学习（RL）面临的主要挑战。解决这些问题的有效方法之一是引入离线数据，例如来自人类专家的先验轨迹或次优探索策略。以往的方法需要大量修改和额外复杂度来确保这些数据的有效利用。本研究提出疑问：我们能否直接应用现有离策略方法来利用在线学习中的离线数据？实验表明，答案是肯定的——但需要对现有离策略RL算法进行最小化但关键性的调整，才能实现可靠性能。我们通过大量消融实验揭示影响性能的关键因素，并提出一套可直接应用的实践建议，无论数据来自少量专家演示还是海量次优轨迹。研究表明，正确应用这些简单建议可在多种竞争性基准测试中实现相较现有方法$\mathbf{2.5\times}$的性能提升，且无需额外计算开销。相关代码已开源在https://github.com/ikostrikov/rlpd。