Building open-ended agents that can autonomously discover a diversity of behaviours is one of the long-standing goals of artificial intelligence. This challenge can be studied in the framework of autotelic RL agents, i.e. agents that learn by selecting and pursuing their own goals, self-organizing a learning curriculum. Recent work identified language has a key dimension of autotelic learning, in particular because it enables abstract goal sampling and guidance from social peers for hindsight relabelling. Within this perspective, we study the following open scientific questions: What is the impact of hindsight feedback from a social peer (e.g. selective vs. exhaustive)? How can the agent learn from very rare language goal examples in its experience replay? How can multiple forms of exploration be combined, and take advantage of easier goals as stepping stones to reach harder ones? To address these questions, we use ScienceWorld, a textual environment with rich abstract and combinatorial physics. We show the importance of selectivity from the social peer's feedback; that experience replay needs to over-sample examples of rare goals; and that following self-generated goal sequences where the agent's competence is intermediate leads to significant improvements in final performance.

翻译：构建能够自主发现多样化行为的开放式智能体是人工智能的长期目标之一。这一挑战可在自驱型强化学习智能体框架下进行研究，即智能体通过选择并追求自身目标来学习，从而自组织学习课程。近期研究指出，语言是自驱型学习的关键维度，特别是因为它能够实现抽象目标采样以及从社交同伴处获得用于事后重新标注的引导。基于这一视角，我们研究了以下开放科学问题：社交同伴的事后反馈（例如选择性反馈与详尽反馈）有何影响？智能体如何从经验回放中极为罕见的语言目标示例中学习？多种探索形式如何结合，并利用较易目标作为达成更难目标的垫脚石？为解决这些问题，我们采用ScienceWorld这一富含抽象与组合物理机制的文本环境。研究表明：社交同伴反馈的选择性至关重要；经验回放需对罕见目标示例进行过采样；遵循智能体能力处于中间水平的自生成目标序列，可显著提升最终性能。