Building a generalist agent that can interact with the world is the intriguing target of AI systems, thus spurring the research for embodied navigation, where an agent is required to navigate according to instructions or respond to queries. Despite the major progress attained, previous works primarily focus on task-specific agents and lack generalizability to unseen scenarios. Recently, LLMs have presented remarkable capabilities across various fields, and provided a promising opportunity for embodied navigation. Drawing on this, we propose the first generalist model for embodied navigation, NaviLLM. It adapts LLMs to embodied navigation by introducing schema-based instruction. The schema-based instruction flexibly casts various tasks into generation problems, thereby unifying a wide range of tasks. This approach allows us to integrate diverse data sources from various datasets into the training, equipping NaviLLM with a wide range of capabilities required by embodied navigation. We conduct extensive experiments to evaluate the performance and generalizability of our model. The experimental results demonstrate that our unified model achieves state-of-the-art performance on CVDN, SOON, and ScanQA. Specifically, it surpasses the previous stats-of-the-art method by a significant margin of 29% in goal progress on CVDN. Moreover, our model also demonstrates strong generalizability and presents impressive results on unseen tasks, e.g., embodied question answering and 3D captioning.

翻译：构建能够与世界交互的通用智能体是人工智能系统的诱人目标，由此催生了体感导航领域的研究，其中智能体需遵循指令导航或响应查询。尽管已取得重大进展，但先前研究主要聚焦于任务特定型智能体，缺乏对未见场景的泛化能力。近期，大语言模型在各领域展现出卓越能力，为体感导航提供了颇具前景的契机。基于此，我们提出首个体感导航通用模型NaviLLM，通过引入基于模式的指令将LLM适配至体感导航领域。基于模式的指令灵活地将各类任务转化为生成问题，从而统一了广泛的任务范畴。该方法使我们能整合不同数据集中的多元数据源进行训练，赋予NaviLLM体感导航所需的全面能力。我们开展了大量实验评估模型的性能与泛化能力。实验结果表明，我们的统一模型在CVDN、SOON及ScanQA基准上均达到最优性能。具体而言，在CVDN任务中，本模型在目标进度指标上以29%的显著优势超越先前最优方法。此外，该模型还展现出强大的泛化能力，在体感问答与3D描述生成等未见任务上取得令人瞩目的成果。