Go-Explore is a powerful family of algorithms designed to solve hard-exploration problems built on the principle of archiving discovered states, and iteratively returning to and exploring from the most promising states. This approach has led to superhuman performance across a wide variety of challenging problems including Atari games and robotic control, but requires manually designing heuristics to guide exploration (i.e., determine which states to save and explore from, and what actions to consider next), which is time-consuming and infeasible in general. To resolve this, we propose Intelligent Go-Explore (IGE) which greatly extends the scope of the original Go-Explore by replacing these handcrafted heuristics with the intelligence and internalized human notions of interestingness captured by giant pretrained foundation models (FMs). This provides IGE with a human-like ability to instinctively identify how interesting or promising any new state is (e.g., discovering new objects, locations, or behaviors), even in complex environments where heuristics are hard to define. Moreover, IGE offers the exciting opportunity to recognize and capitalize on serendipitous discoveries-states encountered during exploration that are valuable in terms of exploration, yet where what makes them interesting was not anticipated by the human user. We evaluate our algorithm on a diverse range of language and vision-based tasks that require search and exploration. Across these tasks, IGE strongly exceeds classic reinforcement learning and graph search baselines, and also succeeds where prior state-of-the-art FM agents like Reflexion completely fail. Overall, Intelligent Go-Explore combines the tremendous strengths of FMs and the powerful Go-Explore algorithm, opening up a new frontier of research into creating more generally capable agents with impressive exploration capabilities.

翻译：Go-Explore是一类基于存档已发现状态、并迭代式返回最有希望状态继续探索原则而设计的强大算法族，用于解决困难探索问题。该方法已在包括Atari游戏和机器人控制在内的多种挑战性问题中实现了超人类性能，但需要手动设计启发式策略来引导探索（即决定保存哪些状态、从何处探索以及下一步考虑哪些动作），这通常耗时且难以通用化。为解决此问题，我们提出智能Go-Explore（IGE），通过用巨型预训练基础模型（FMs）所具备的智能及其内化的人类兴趣认知来替代这些人工设计的启发式策略，从而极大拓展了原始Go-Explore的适用范围。这使得IGE具备类人的本能能力，能够即时识别任何新状态的有趣程度或潜力（例如发现新物体、位置或行为），即使在难以定义启发式规则的复杂环境中亦然。此外，IGE提供了识别并利用意外发现的激动可能性——这些在探索过程中遇到的状态对探索具有重要价值，但其有趣特性超出用户预期。我们在多种需要搜索与探索的语言和视觉任务上评估了算法性能。在所有任务中，IGE显著超越了经典强化学习和图搜索基线，并在Reflexion等先前最先进的FM智能体完全失败的任务上取得成功。总体而言，智能Go-Explore融合了基础模型的强大能力与Go-Explore算法的优势，为创建具有卓越探索能力的通用智能体开辟了新的研究前沿。