The rise of large foundation models, trained on extensive datasets, is revolutionizing the field of AI. Models such as SAM, DALL-E2, and GPT-4 showcase their adaptability by extracting intricate patterns and performing effectively across diverse tasks, thereby serving as potent building blocks for a wide range of AI applications. Autonomous driving, a vibrant front in AI applications, remains challenged by the lack of dedicated vision foundation models (VFMs). The scarcity of comprehensive training data, the need for multi-sensor integration, and the diverse task-specific architectures pose significant obstacles to the development of VFMs in this field. This paper delves into the critical challenge of forging VFMs tailored specifically for autonomous driving, while also outlining future directions. Through a systematic analysis of over 250 papers, we dissect essential techniques for VFM development, including data preparation, pre-training strategies, and downstream task adaptation. Moreover, we explore key advancements such as NeRF, diffusion models, 3D Gaussian Splatting, and world models, presenting a comprehensive roadmap for future research. To empower researchers, we have built and maintained https://github.com/zhanghm1995/Forge_VFM4AD, an open-access repository constantly updated with the latest advancements in forging VFMs for autonomous driving.

翻译：大规模基础模型的兴起，依托于海量数据集的训练，正在彻底改变人工智能领域。像SAM、DALL-E2和GPT-4这样的模型，通过提取复杂模式并在多种任务上有效执行，展示了其适应性，从而成为广泛AI应用的有力构建模块。自动驾驶作为人工智能应用中一个充满活力的前沿领域，仍面临缺乏专用视觉基础模型（VFM）的挑战。全面训练数据的稀缺、多传感器集成的需求以及多样化的任务特定架构，为该领域视觉基础模型的发展设置了重大障碍。本文深入探讨了为自动驾驶专门构建视觉基础模型的关键挑战，同时勾勒了未来方向。通过对超过250篇论文的系统分析，我们剖析了视觉基础模型开发中的关键技术，包括数据准备、预训练策略以及下游任务适配。此外，我们探讨了诸如NeRF、扩散模型、3D高斯平铺以及世界模型等关键进展，为未来研究提供了全面的路线图。为助力研究者，我们构建并维护了https://github.com/zhanghm1995/Forge_VFM4AD，这是一个持续更新面向自动驾驶视觉基础模型构建最新进展的开放获取知识库。