Vision-and-Language Navigation (VLN) stands as a key research problem of Embodied AI, aiming at enabling agents to navigate in unseen environments following linguistic instructions. In this field, generalization is a long-standing challenge, either to out-of-distribution scenes or from Sim to Real. In this paper, we propose NaVid, a video-based large vision language model (VLM), to mitigate such a generalization gap. NaVid makes the first endeavour to showcase the capability of VLMs to achieve state-of-the-art level navigation performance without any maps, odometer and depth inputs. Following human instruction, NaVid only requires an on-the-fly video stream from a monocular RGB camera equipped on the robot to output the next-step action. Our formulation mimics how humans navigate and naturally gets rid of the problems introduced by odometer noises, and the Sim2Real gaps from map or depth inputs. Moreover, our video-based approach can effectively encode the historical observations of robots as spatio-temporal contexts for decision-making and instruction following. We train NaVid with 550k navigation samples collected from VLN-CE trajectories, including action-planning and instruction-reasoning samples, along with 665k large-scale web data. Extensive experiments show that NaVid achieves SOTA performance in simulation environments and the real world, demonstrating superior cross-dataset and Sim2Real transfer. We thus believe our proposed VLM approach plans the next step for not only the navigation agents but also this research field.

翻译：视觉语言导航（VLN）是具身智能领域的关键研究问题，旨在使智能体能够根据语言指令在未知环境中导航。在该领域中，泛化能力（无论是针对分布外场景还是从仿真到真实的迁移）始终是一项长期挑战。本文提出NaVid——一种基于视频的大型视觉语言模型（VLM），以缩小此类泛化差距。NaVid首次尝试展示了视觉语言模型在无需地图、里程计和深度输入的情况下实现顶尖导航性能的能力。遵循人类指令，NaVid仅需机器人配备的单目RGB摄像头实时采集视频流，即可输出下一步动作。我们的设计方案模拟人类导航方式，自然消除了里程计噪声引入的问题以及地图或深度输入带来的仿真到现实（Sim2Real）差距。此外，基于视频的方法可以有效地将机器人历史观测编码为时空上下文，用于决策制定和指令跟随。我们利用从VLN-CE轨迹中收集的55万条导航样本（包括动作规划与指令推理样本）以及66.5万条大规模网络数据对NaVid进行训练。大量实验表明，NaVid在仿真环境和真实世界中均达到当前最优（SOTA）性能，展现出卓越的跨数据集与Sim2Real迁移能力。因此，我们相信所提出的VLM方法不仅为导航智能体，也为整个研究领域规划了下一步方向。