Sample efficiency and exploration remain major challenges in online reinforcement learning (RL). A powerful approach that can be applied to address these issues is the inclusion of offline data, such as prior trajectories from a human expert or a sub-optimal exploration policy. Previous methods have relied on extensive modifications and additional complexity to ensure the effective use of this data. Instead, we ask: can we simply apply existing off-policy methods to leverage offline data when learning online? In this work, we demonstrate that the answer is yes; however, a set of minimal but important changes to existing off-policy RL algorithms are required to achieve reliable performance. We extensively ablate these design choices, demonstrating the key factors that most affect performance, and arrive at a set of recommendations that practitioners can readily apply, whether their data comprise a small number of expert demonstrations or large volumes of sub-optimal trajectories. We see that correct application of these simple recommendations can provide a $\mathbf{2.5\times}$ improvement over existing approaches across a diverse set of competitive benchmarks, with no additional computational overhead.

翻译：样本效率与探索仍是在线强化学习（RL）中的主要挑战。一种可有效应对这些问题的强大方法是引入离线数据，例如来自人类专家或次优探索策略的先前轨迹。以往的方法依赖大量修改和额外复杂性来确保这些数据的有效利用。相反，我们提出疑问：是否可以直接应用现有的离策略方法，在在线学习时利用离线数据？在本工作中，我们证明答案是肯定的；但需对现有离策略强化学习算法进行一组最小而重要的改动，以实现可靠的性能。我们广泛消融了这些设计选择，揭示了影响性能的关键因素，并得出一套实践者可立即应用的建议——无论其数据包含少量专家演示还是大量次优轨迹。我们看到，正确应用这些简单建议能在多样化的竞争性基准测试中，相比现有方法实现$\mathbf{2.5\times}$的性能提升，且无额外计算开销。