General-purpose pre-trained models ("foundation models") have enabled practitioners to produce generalizable solutions for individual machine learning problems with datasets that are significantly smaller than those required for learning from scratch. Such models are typically trained on large and diverse datasets with weak supervision, consuming much more training data than is available for any individual downstream application. In this paper, we describe the Visual Navigation Transformer (ViNT), a foundation model that aims to bring the success of general-purpose pre-trained models to vision-based robotic navigation. ViNT is trained with a general goal-reaching objective that can be used with any navigation dataset, and employs a flexible Transformer-based architecture to learn navigational affordances and enable efficient adaptation to a variety of downstream navigational tasks. ViNT is trained on a number of existing navigation datasets, comprising hundreds of hours of robotic navigation from a variety of different robotic platforms, and exhibits positive transfer, outperforming specialist models trained on singular datasets. ViNT can be augmented with diffusion-based subgoal proposals to explore novel environments, and can solve kilometer-scale navigation problems when equipped with long-range heuristics. ViNT can also be adapted to novel task specifications with a technique inspired by prompt-tuning, where the goal encoder is replaced by an encoding of another task modality (e.g., GPS waypoints or routing commands) embedded into the same space of goal tokens. This flexibility and ability to accommodate a variety of downstream problem domains establishes ViNT as an effective foundation model for mobile robotics. For videos, code, and model checkpoints, see our project page at https://visualnav-transformer.github.io.

翻译：通用预训练模型（"基础模型"）使从业者能够通过显著少于从零学习所需的数据集，为单个机器学习问题生成可泛化的解决方案。此类模型通常利用弱监督在大型多样化数据集上训练，消耗的训练数据远超任何特定下游应用所能获得的规模。本文提出视觉导航Transformer（ViNT），这一基础模型旨在将通用预训练模型成功经验引入基于视觉的机器人导航领域。ViNT通过可适配任意导航数据集的通用目标到达目标进行训练，采用灵活的Transformer架构学习导航可行动作，并实现对多种下游导航任务的高效适配。该模型在多个现有导航数据集上训练，涵盖不同机器人平台数百小时的导航数据，展现出正向迁移能力——其性能优于在单一数据集上训练的专家模型。ViNT可结合基于扩散模型的子目标提议探索未知环境，并在配备长程启发式策略时解决千米级导航问题。此外，受提示调优技术启发，ViNT可通过替换目标编码器为其他任务模态编码（如GPS路径点或路由指令）并嵌入相同目标标记空间，进而适配新型任务规范。这种灵活性与跨下游领域适应性，使ViNT成为移动机器人领域有效的基础模型。相关视频、代码及模型检查点详见项目主页：https://visualnav-transformer.github.io。