Open-ended algorithms aim to learn new, interesting behaviors forever. That requires a vast environment search space, but there are thus infinitely many possible tasks. Even after filtering for tasks the current agent can learn (i.e., learning progress), countless learnable yet uninteresting tasks remain (e.g., minor variations of previously learned tasks). An Achilles Heel of open-endedness research is the inability to quantify (and thus prioritize) tasks that are not just learnable, but also $\textit{interesting}$ (e.g., worthwhile and novel). We propose solving this problem by $\textit{Open-endedness via Models of human Notions of Interestingness}$ (OMNI). The insight is that we can utilize large (language) models (LMs) as a model of interestingness (MoI), because they $\textit{already}$ internalize human concepts of interestingness from training on vast amounts of human-generated data, where humans naturally write about what they find interesting or boring. We show that LM-based MoIs improve open-ended learning by focusing on tasks that are both learnable $\textit{and interesting}$, outperforming baselines based on uniform task sampling or learning progress alone. This approach has the potential to dramatically advance the ability to intelligently select which tasks to focus on next (i.e., auto-curricula), and could be seen as AI selecting its own next task to learn, facilitating self-improving AI and AI-Generating Algorithms.

翻译：开放式算法旨在持续学习新的、有趣的行为。这需要巨大的环境搜索空间，但其中存在无限可能的任务。即使过滤出当前智能体可学习的任务（即学习进展），仍会留下无数可学习却无趣的任务（例如，已学任务的微小变体）。开放式研究的致命弱点在于无法量化（从而无法优先级排序）那些不仅可学习、而且具有"有趣性"（例如，值得学习的和新颖的）的任务。我们提出通过"基于人类兴趣模型的开放式学习"（OMNI）来解决这一问题。其核心洞察在于：我们可以利用大型（语言）模型作为兴趣模型（MoI），因为它们在基于海量人类生成数据的训练过程中，已经内化了人类对兴趣与无聊的概念——人类自然会在数据中描述令其感兴趣或无聊的内容。我们证明，基于语言模型的兴趣模型能够通过聚焦既"可学习"又"有趣"的任务来改进开放式学习，其性能优于基于均匀任务采样或仅依赖学习进展的基线方法。这一方法有望显著提升智能选择下一关注任务的能力（即自动课程学习），并可被视为人工智能主动选择自身需学习的下一任务，从而促进自我改进型人工智能与人工智能生成算法的发展。