General-purpose pre-trained models ("foundation models") have enabled practitioners to produce generalizable solutions for individual machine learning problems with datasets that are significantly smaller than those required for learning from scratch. Such models are typically trained on large and diverse datasets with weak supervision, consuming much more training data than is available for any individual downstream application. In this paper, we describe the Visual Navigation Transformer (ViNT), a foundation model that aims to bring the success of general-purpose pre-trained models to vision-based robotic navigation. ViNT is trained with a general goal-reaching objective that can be used with any navigation dataset, and employs a flexible Transformer-based architecture to learn navigational affordances and enable efficient adaptation to a variety of downstream navigational tasks. ViNT is trained on a number of existing navigation datasets, comprising hundreds of hours of robotic navigation from a variety of different robotic platforms, and exhibits positive transfer, outperforming specialist models trained on singular datasets. ViNT can be augmented with diffusion-based subgoal proposals to explore novel environments, and can solve kilometer-scale navigation problems when equipped with long-range heuristics. ViNT can also be adapted to novel task specifications with a technique inspired by prompt-tuning, where the goal encoder is replaced by an encoding of another task modality (e.g., GPS waypoints or routing commands) embedded into the same space of goal tokens. This flexibility and ability to accommodate a variety of downstream problem domains establishes ViNT as an effective foundation model for mobile robotics. For videos, code, and model checkpoints, see our project page at https://visualnav-transformer.github.io.

翻译：通用预训练模型（"基础模型"）使得从业者能够为单个机器学习问题生成泛化解，且所需数据集远小于从头训练所需的规模。此类模型通常在大规模多样化数据集上通过弱监督训练，消耗的训练数据远超任何下游应用场景的可用数据量。本文提出视觉导航Transformer（ViNT），这是一个旨在将通用预训练模型的成功经验引入基于视觉的机器人导航领域的基础模型。ViNT采用可适配任意导航数据集的通用目标达成目标进行训练，并基于灵活的Transformer架构学习导航可供性，从而实现对多种下游导航任务的高效适配。ViNT基于多个现有导航数据集训练，涵盖来自不同机器人平台的数百小时机器人导航数据，展现出正向迁移特性，性能优于在单一数据集上训练的专业模型。通过扩散模型驱动的子目标提议机制，ViNT可探索未知环境；配备长程启发式策略后，其能解决公里级导航问题。ViNT还可通过类似提示微调的技术适配新型任务规范：将目标编码器替换为另一任务模态（如GPS航点或路由指令）的编码，并将其嵌入与目标标记相同的表征空间。这种灵活性与适配多种下游领域的能力，使ViNT成为移动机器人领域高效的基础模型。相关视频、代码及模型检查点请访问项目页面：https://visualnav-transformer.github.io。