We study the problem of building a controller that can follow open-ended instructions in open-world environments. We propose to follow reference videos as instructions, which offer expressive goal specifications while eliminating the need for expensive text-gameplay annotations. A new learning framework is derived to allow learning such instruction-following controllers from gameplay videos while producing a video instruction encoder that induces a structured goal space. We implement our agent GROOT in a simple yet effective encoder-decoder architecture based on causal transformers. We evaluate GROOT against open-world counterparts and human players on a proposed Minecraft SkillForge benchmark. The Elo ratings clearly show that GROOT is closing the human-machine gap as well as exhibiting a 70% winning rate over the best generalist agent baseline. Qualitative analysis of the induced goal space further demonstrates some interesting emergent properties, including the goal composition and complex gameplay behavior synthesis. Code and video can be found on the website https://craftjarvis-groot.github.io.

翻译：摘要：本文研究在开放世界环境中构建能够遵循开放式指令的控制模型。我们提出将参考视频作为指令，既能提供富有表达力的目标规范，又无需昂贵的文本-游戏标注。本文推导出一种新的学习框架，使得控制器可以从游戏视频中学习遵循指令，同时生成一个视频指令编码器，诱导出结构化的目标空间。我们基于因果Transformer，以简洁高效的编码器-解码器架构实现了智能体GROOT。在提出的Minecraft SkillForge基准测试中，我们将GROOT与开放世界同类模型及人类玩家进行了对比。ELO评分表明，GROOT正在缩小人机差距，并且相较于最佳通用智能体基线，展现出70%的胜率。对诱导出的目标空间进行定性分析进一步揭示了一些有趣的涌现特性，包括目标组合与复杂游戏行为合成。代码与视频可在网站https://craftjarvis-groot.github.io获取。