Despite advancements in Large Language Models (LLMs) and Large Multimodal Models (LMMs), their integration into language-grounded, human-like embodied agents remains incomplete, hindering complex real-life task performance in physical environments. Existing integrations often feature limited open sourcing, challenging collective progress in this field. We introduce LEGENT, an open, scalable platform for developing embodied agents using LLMs and LMMs. LEGENT offers a dual approach: a rich, interactive 3D environment with communicable and actionable agents, paired with a user-friendly interface, and a sophisticated data generation pipeline utilizing advanced algorithms to exploit supervision from simulated worlds at scale. In our experiments, an embryonic vision-language-action model trained on LEGENT-generated data surpasses GPT-4V in embodied tasks, showcasing promising generalization capabilities.

翻译：尽管大语言模型（LLMs）和多模态大模型（LMMs）取得了显著进展，但其与基于语言的人类化具身智能体的集成仍不完整，阻碍了其在物理环境中执行复杂现实任务的能力。现有集成方案普遍存在开源程度有限的问题，制约了该领域的协同发展。我们提出LEGENT——一个开放、可扩展的具身智能体开发平台，可基于LLMs和LMMs构建智能体。该平台采用双轨设计：一方面提供包含可通信、可行动智能体的丰富交互式3D环境及用户友好界面，另一方面构建了利用先进算法的数据生成管线，能够大规模挖掘模拟环境中的监督信号。实验中，基于LEGENT生成数据训练的初期视觉-语言-动作模型在具身任务中已超越GPT-4V，展现出显著的泛化潜力。