The process of discovery requires active exploration -- the act of collecting new and informative data. However, efficient autonomous exploration remains a major unsolved problem. The dominant paradigm addresses this challenge by using Reinforcement Learning (RL) to train agents with intrinsic motivation, maximizing a composite objective of extrinsic and intrinsic rewards. We suggest that this approach incurs unnecessary overhead: while policy optimization is necessary for precise task execution, employing such machinery solely to expand state coverage may be inefficient. In this paper, we propose a new paradigm that explicitly separates exploration from exploitation and bypasses RL during the exploration phase. Our method uses a tree-search strategy inspired by the Go-With-The-Winner algorithm, paired with a measure of epistemic uncertainty to systematically drive exploration. By removing the overhead of policy optimization, our approach explores an order of magnitude more efficiently than standard intrinsic motivation baselines on hard Atari benchmarks. Further, we demonstrate that the discovered trajectories can be distilled into deployable policies using existing supervised backward learning algorithms, achieving state-of-the-art scores by a wide margin on Montezuma's Revenge, Pitfall!, and Venture without relying on domain-specific knowledge. Finally, we demonstrate the generality of our framework in high-dimensional continuous action spaces by solving the MuJoCo Adroit dexterous manipulation and AntMaze tasks in a sparse-reward setting, directly from image observations and without expert demonstrations or offline datasets. To the best of our knowledge, this has not been achieved before for the Adroit tasks.

翻译：发现过程需要主动探索——即收集新颖且具有信息量的数据。然而，高效的自主探索仍是一个未解决的核心难题。当前主流范式通过强化学习（RL）训练具有内在动机的智能体，将外在奖励与内在奖励的复合目标最大化来解决此问题。我们认为该方法存在不必要的开销：尽管策略优化对于精确任务执行至关重要，但仅为了扩展状态覆盖范围而采用此类机制可能效率低下。本文提出一种新范式，明确将探索与利用分离，并在探索阶段绕过强化学习。我们的方法采用受"跟随胜者算法"启发的树搜索策略，结合认知不确定性度量来系统性地驱动探索。通过消除策略优化的开销，该方法在困难Atari基准测试中的探索效率比标准内在动机基线高出一个数量级。进一步，我们证明发现轨迹可通过现有监督反向学习算法蒸馏为可部署策略，在无需领域特定知识的情况下，于Montezuma's Revenge、Pitfall!和Venture任务中大幅超越现有最优分数。最后，我们通过解决稀疏奖励设置下的MuJoCo Adroit灵巧操控和AntMaze任务，展示了该方法在高维连续动作空间中的普适性——这些任务直接基于图像观测，无需专家示范或离线数据集。据我们所知，Adroit任务此前尚未实现此类突破。