Navigation foundation models trained on massive web-scale data enable agents to generalize across diverse environments and embodiments. However, these models, which are trained solely on offline data, often lack the capacity to reason about the consequences of their actions or adapt through counterfactual understanding. They thus face significant limitations in real-world urban navigation, where interactive and safe behaviors, such as avoiding obstacles and moving pedestrians, are critical. To tackle these challenges, we introduce the Seeing-to-Experiencing (S2E) learning framework to scale the capability of navigation foundation models with reinforcement learning. S2E combines the strengths of pretraining on offline videos and post-training through reinforcement learning. It maintains the model's generalizability acquired from large-scale real-world videos while enhancing its interactivity through reinforcement learning in simulation environments. Specifically, we introduce two innovations: (1) an Anchor-Guided Distribution Matching strategy for offline pretraining, which stabilizes learning and models diverse motion patterns through anchor-based supervision; and (2) a Residual-Attention Module for reinforcement learning, which obtains reactive behaviors from simulation environments without erasing the model's pretrained knowledge. Moreover, we establish a comprehensive end-to-end evaluation benchmark, NavBench-GS, built on photorealistic 3D Gaussian Splatting reconstructions of real-world scenes that incorporate physical interactions. It can systematically assess the generalizability and safety of navigation foundation models.

翻译：基于大规模网络数据训练的导航基础模型使智能体能够泛化至不同环境与形态。然而，这些仅依赖离线数据训练的模型往往缺乏对行为后果的推理能力，也无法通过反事实理解进行适应。因此在真实城市导航场景中面临显著局限——此类环境中，如规避障碍物与移动行人等交互式安全行为至关重要。为应对这些挑战，我们提出"看见到体验"(S2E)学习框架，通过强化学习扩展导航基础模型的能力。S2E融合了离线视频预训练与强化学习后训练的优势，在保持模型从大规模真实世界视频获得的泛化能力的同时，通过仿真环境中的强化学习增强其交互性。具体而言，我们引入两项创新：(1) 用于离线预训练的锚点引导分布匹配策略，通过基于锚点的监督稳定学习过程并建模多样化运动模式；(2) 用于强化学习的残差注意力模块，在保留模型预训练知识的前提下从仿真环境获取反应性行为。此外，我们基于融入物理交互的真实场景光场重建技术，建立了端到端综合评估基准NavBench-GS，可系统评估导航基础模型的泛化能力与安全性。