The process of discovery requires active exploration -- the act of collecting new and informative data. However, efficient autonomous exploration remains a major unsolved problem. The dominant paradigm addresses this challenge by using Reinforcement Learning (RL) to train agents with intrinsic motivation, maximizing a composite objective of extrinsic and intrinsic rewards. We suggest that this approach incurs unnecessary overhead: while policy optimization is necessary for precise task execution, employing such machinery solely to expand state coverage may be inefficient. In this paper, we propose a new paradigm that explicitly separates exploration from exploitation and bypasses RL during the exploration phase. Our method uses a tree-search strategy inspired by the Go-With-The-Winner algorithm, paired with a measure of epistemic uncertainty to systematically drive exploration. By removing the overhead of policy optimization, our approach explores an order of magnitude more efficiently than standard intrinsic motivation baselines on hard Atari benchmarks. Further, we demonstrate that the discovered trajectories can be distilled into deployable policies using existing supervised backward learning algorithms, achieving state-of-the-art scores by a wide margin on Montezuma's Revenge, Pitfall!, and Venture without relying on domain-specific knowledge. Finally, we demonstrate the generality of our framework in high-dimensional continuous action spaces by solving the MuJoCo Adroit dexterous manipulation and AntMaze tasks in a sparse-reward setting, directly from image observations and without expert demonstrations or offline datasets. To the best of our knowledge, this has not been achieved before.

翻译：发现过程需要主动探索——即收集新颖且信息丰富的数据。然而，高效的自主探索仍是一个尚未解决的主要难题。当前主流范式通过使用强化学习训练具有内在动机的智能体来应对这一挑战，其目标是最大化由外在奖励和内在奖励组成的复合目标函数。我们认为，这种方法引入了不必要的计算开销：虽然策略优化对于精确的任务执行是必要的，但仅仅为了扩展状态覆盖范围而使用此类机制可能效率低下。本文提出了一种新范式，明确地将探索与利用分离，并在探索阶段绕过强化学习。我们的方法采用受"随胜者同行"算法启发的树搜索策略，结合认知不确定性度量来系统性地驱动探索。通过移除策略优化的计算开销，我们的方法在困难的Atari基准测试中，探索效率比标准内在动机基线方法高出一个数量级。此外，我们证明，使用现有的监督反向学习算法可以将发现的轨迹提炼为可部署的策略，在Montezuma's Revenge、Pitfall!和Venture等任务中，在不依赖领域特定知识的情况下，以显著优势取得了当前最优得分。最后，我们通过在稀疏奖励设定下，直接从图像观测出发，且不使用专家演示或离线数据集，解决了MuJoCo Adroit灵巧操作和AntMaze任务，证明了我们的框架在高维连续动作空间中的通用性。据我们所知，这是此前尚未实现过的。