Humans are excellent at understanding language and vision to accomplish a wide range of tasks. In contrast, creating general instruction-following embodied agents remains a difficult challenge. Prior work that uses pure language-only models lack visual grounding, making it difficult to connect language instructions with visual observations. On the other hand, methods that use pre-trained multimodal models typically come with divided language and visual representations, requiring designing specialized network architecture to fuse them together. We propose a simple yet effective model for robots to solve instruction-following tasks in vision-based environments. Our \ours method consists of a multimodal transformer that encodes visual observations and language instructions, and a transformer-based policy that predicts actions based on encoded representations. The multimodal transformer is pre-trained on millions of image-text pairs and natural language text, thereby producing generic cross-modal representations of observations and instructions. The transformer-based policy keeps track of the full history of observations and actions, and predicts actions autoregressively. Despite its simplicity, we show that this unified transformer model outperforms all state-of-the-art pre-trained or trained-from-scratch methods in both single-task and multi-task settings. Our model also shows better model scalability and generalization ability than prior work.

翻译：人类擅长通过理解语言和视觉信息完成各类任务，但构建通用的指令遵循具身智能体仍是严峻挑战。使用纯语言模型的现有方法缺乏视觉基础，难以将语言指令与视觉观测关联起来。而采用预训练多模态模型的方法通常存在语言与视觉表征分离的问题，需设计专门的网络架构进行融合。本文提出一种简洁高效的机器人模型，用于解决基于视觉环境中的指令遵循任务。我们的方法包含两个核心组件：用于编码视觉观测和语言指令的多模态Transformer，以及基于Transformer的策略网络——后者根据编码表征预测动作。多模态Transformer通过在数百万图像-文本对及自然语言文本上预训练，生成通用的观测-指令跨模态表征。Transformer策略网络则跟踪完整的观测-动作历史，以自回归方式预测动作。尽管架构简洁，但实验证明，这种统一Transformer模型在单任务和多任务场景中均全面超越现有最优的预训练或从头训练方法。此外，该模型展现出比先前工作更优异的可扩展性和泛化能力。