Developing agents capable of open-endedly discovering and learning novel skills is a grand challenge in Artificial Intelligence. While reinforcement learning offers a powerful framework for training agents to master complex skills, it typically relies on hand-designed reward functions. This is infeasible for open-ended skill discovery, where the set of meaningful skills is not known a priori. While recent methods have shown promising results towards automating reward function design, they remain limited to refining rewards for pre-defined tasks. To address this limitation, we introduce Continuous Open-ended Discovery and Evolution of Skills as Hierarchical Reward Programs (CODE-SHARP), a novel framework leveraging Foundation Models (FM) to open-endedly expand and refine a hierarchical skill archive, structured as a directed graph of executable reward functions in code. We show that a goal-conditioned agent trained exclusively on the rewards generated by the discovered SHARP skills learns to solve increasingly long-horizon goals in the Craftax environment. When composed by a high-level FM-based planner, the discovered skills enable a single goal-conditioned agent to solve complex, long-horizon tasks, outperforming both pretrained agents and task-specific expert policies by over $134$% on average. We will open-source our code and provide additional videos $\href{https://sites.google.com/view/code-sharp/homepage}{here}$.

翻译：开发能够以开放式方式发现和学习新技能的智能体是人工智能领域的一项重大挑战。虽然强化学习为训练智能体掌握复杂技能提供了一个强大的框架，但它通常依赖于人工设计的奖励函数。这对于开放式技能发现而言是不可行的，因为其中有意义的技能集合并非先验已知。尽管近期的方法在自动化奖励函数设计方面已显示出有前景的结果，但它们仍局限于为预定义任务优化奖励。为了应对这一局限，我们引入了“作为分层奖励程序的持续开放式技能发现与演化”（CODE-SHARP），这是一个新颖的框架，它利用基础模型（FM）来开放式地扩展和优化一个分层技能档案库，该档案库被结构化为一个由可执行的代码化奖励函数组成的有向图。我们证明，一个仅通过所发现的SHARP技能生成的奖励进行训练的目标条件智能体，能够学会在Craftax环境中解决日益增长的长时程目标。当由一个基于高层FM的规划器进行组合时，所发现的技能使得单个目标条件智能体能够解决复杂的长时程任务，其平均性能分别超过预训练智能体和任务特定专家策略超过$134$%。我们将开源我们的代码，并在此处提供更多视频 $\href{https://sites.google.com/view/code-sharp/homepage}{here}$。