Learning to solve tasks from a sparse reward signal is a major challenge for standard reinforcement learning (RL) algorithms. However, in the real world, agents rarely need to solve sparse reward tasks entirely from scratch. More often, we might possess prior experience to draw on that provides considerable guidance about which actions and outcomes are possible in the world, which we can use to explore more effectively for new tasks. In this work, we study how prior data without reward labels may be used to guide and accelerate exploration for an agent solving a new sparse reward task. We propose a simple approach that learns a reward model from online experience, labels the unlabeled prior data with optimistic rewards, and then uses it concurrently alongside the online data for downstream policy and critic optimization. This general formula leads to rapid exploration in several challenging sparse-reward domains where tabula rasa exploration is insufficient, including the AntMaze domain, Adroit hand manipulation domain, and a visual simulated robotic manipulation domain. Our results highlight the ease of incorporating unlabeled prior data into existing online RL algorithms, and the (perhaps surprising) effectiveness of doing so.

翻译：从稀疏奖励信号中学习解决任务是标准强化学习（RL）算法面临的主要挑战。然而在现实世界中，智能体很少需要完全从零开始解决稀疏奖励任务。更常见的情况是，我们可能拥有可借鉴的先验经验，这些经验为世界中可能采取的动作和产生的结果提供了重要指导，从而帮助我们在新任务中更有效地进行探索。本研究探讨如何利用无奖励标签的先验数据，引导并加速智能体解决新稀疏奖励任务的探索过程。我们提出一种简洁方法：从在线经验中学习奖励模型，为无标签先验数据赋予乐观奖励，随后将其与在线数据并行用于下游策略与评价器的优化。这一通用公式在多个具有挑战性的稀疏奖励领域（包括AntMaze域、Adroit手部操控域以及视觉模拟机器人操控域）中实现了快速探索，而纯白板式探索在这些场景中效果不足。我们的研究结果凸显了将无标签先验数据融入现有在线强化学习算法的简便性及其（或许令人惊讶的）有效性。