Building a generalist agent that can interact with the world is the intriguing target of AI systems, thus spurring the research for embodied navigation, where an agent is required to navigate according to instructions or respond to queries. Despite the major progress attained, previous works primarily focus on task-specific agents and lack generalizability to unseen scenarios. Recently, LLMs have presented remarkable capabilities across various fields, and provided a promising opportunity for embodied navigation. Drawing on this, we propose the first generalist model for embodied navigation, NaviLLM. It adapts LLMs to embodied navigation by introducing schema-based instruction. The schema-based instruction flexibly casts various tasks into generation problems, thereby unifying a wide range of tasks. This approach allows us to integrate diverse data sources from various datasets into the training, equipping NaviLLM with a wide range of capabilities required by embodied navigation. We conduct extensive experiments to evaluate the performance and generalizability of our model. The experimental results demonstrate that our unified model achieves state-of-the-art performance on CVDN, SOON, and ScanQA. Specifically, it surpasses the previous stats-of-the-art method by a significant margin of 29% in goal progress on CVDN. Moreover, our model also demonstrates strong generalizability and presents impressive results on unseen tasks, e.g., embodied question answering and 3D captioning.

翻译：构建一个能与世界交互的通用智能体是人工智能系统引人入胜的目标，进而推动了具身导航研究的发展——在该任务中，智能体需根据指令导航或响应查询。尽管已取得重大进展，以往研究主要聚焦于任务特定型智能体，缺乏对未见场景的泛化能力。近年来，大语言模型在各领域展现出卓越能力，为具身导航提供了充满前景的机遇。基于此，我们提出了首个具身导航通用模型NaviLLM。该模型通过引入基于模式的指令，将LLM适配至具身导航任务。基于模式的指令可将各类任务灵活转化为生成问题，从而统一多种任务类型。这种方法使我们能够整合来自不同数据集的多样化数据源进行训练，赋予NaviLLM具身导航所需的广泛能力。我们开展了大量实验以评估模型的性能与泛化能力。实验结果表明，我们的统一模型在CVDN、SOON和ScanQA数据集上达到了最先进性能。具体而言，在CVDN数据集上，其目标进度指标较此前最先进方法显著提升29%。此外，模型还展现出强大的泛化能力，在未见任务（如具身问答与三维描述生成）上呈现出令人印象深刻的结果。