Sample efficiency and exploration remain major challenges in online reinforcement learning (RL). A powerful approach that can be applied to address these issues is the inclusion of offline data, such as prior trajectories from a human expert or a sub-optimal exploration policy. Previous methods have relied on extensive modifications and additional complexity to ensure the effective use of this data. Instead, we ask: can we simply apply existing off-policy methods to leverage offline data when learning online? In this work, we demonstrate that the answer is yes; however, a set of minimal but important changes to existing off-policy RL algorithms are required to achieve reliable performance. We extensively ablate these design choices, demonstrating the key factors that most affect performance, and arrive at a set of recommendations that practitioners can readily apply, whether their data comprise a small number of expert demonstrations or large volumes of sub-optimal trajectories. We see that correct application of these simple recommendations can provide a $\mathbf{2.5\times}$ improvement over existing approaches across a diverse set of competitive benchmarks, with no additional computational overhead.

翻译：样本效率与探索仍是在线强化学习中的主要挑战。一种可应对这些问题的有效方法是将离线数据纳入其中，例如来自人类专家的先前轨迹或次优探索策略。以往的方法依赖于大量修改和额外的复杂性来确保这些数据的有效利用。与此相反，我们提出疑问：在学习在线任务时，能否直接应用现有的离策略方法来利用离线数据？本研究表明答案是肯定的；不过，需要对现有离策略强化学习算法进行一组最小但重要的改动，以实现可靠性能。我们全面消融了这些设计选择，揭示了最影响性能的关键因素，并得出一套从业者可立即应用的建议，无论其数据包含少量专家演示还是大量次优轨迹。我们发现，正确应用这些简单建议可在多样化的竞争基准测试中，相较于现有方法实现$\mathbf{2.5\times}$的性能提升，且无需额外计算开销。