Due to the need to interact with the real world, embodied agents are required to possess comprehensive prior knowledge, long-horizon planning capability, and a swift response speed. Despite recent large language model (LLM) based agents achieving promising performance, they still exhibit several limitations. For instance, the output of LLMs is a descriptive sentence, which is ambiguous when determining specific actions. To address these limitations, we introduce the large auto-regressive model (LARM). LARM leverages both text and multi-view images as input and predicts subsequent actions in an auto-regressive manner. To train LARM, we develop a novel data format named auto-regressive node transmission structure and assemble a corresponding dataset. Adopting a two-phase training regimen, LARM successfully harvests enchanted equipment in Minecraft, which demands significantly more complex decision-making chains than the highest achievements of prior best methods. Besides, the speed of LARM is 6.8x faster.

翻译：由于需要与现实世界交互，具身智能体必须具备全面的先验知识、长视野规划能力以及快速响应速度。尽管近期基于大语言模型（LLM）的智能体取得了令人瞩目的性能，但仍存在若干局限性。例如，LLM的输出为描述性语句，在确定具体动作时存在歧义。为应对这些局限，我们提出了大型自回归模型（LARM）。LARM同时利用文本和多视角图像作为输入，以自回归方式预测后续动作。为训练LARM，我们开发了一种名为自回归节点传输结构的新型数据格式，并构建了相应数据集。通过采用两阶段训练策略，LARM成功在《我的世界》中获取了附魔装备，该任务所需的决策链复杂度远超现有最佳方法的最高成就。此外，LARM的响应速度提升了6.8倍。