With the broader adoption and highly successful development of Large Language Models (LLMs), there has been growing interest and demand for applying LLMs to autonomous driving technology. Driven by their natural language understanding and reasoning capabilities, LLMs have the potential to enhance various aspects of autonomous driving systems, from perception and scene understanding to interactive decision-making. This paper first introduces the novel concept of designing Large Language Models for Autonomous Driving (LLM4AD), followed by a review of existing LLM4AD studies. Then, a comprehensive benchmark is proposed for evaluating the instruction-following and reasoning abilities of LLM4AD systems, which includes LaMPilot-Bench, CARLA Leaderboard 1.0 Benchmark in simulation and NuPlanQA for multi-view visual question answering. Furthermore, extensive real-world experiments are conducted on autonomous vehicle platforms, examining both on-cloud and on-edge LLM deployment for personalized decision-making and motion control. Next, the future trends of integrating language diffusion models into autonomous driving are explored, exemplified by the proposed ViLaD (Vision-Language Diffusion) framework. Finally, the main challenges of LLM4AD are discussed, including latency, deployment, security and privacy, safety, trust and transparency, and personalization.

翻译：随着大语言模型（LLMs）的广泛采用和成功发展，将LLMs应用于自动驾驶技术的兴趣和需求日益增长。受其自然语言理解与推理能力的驱动，LLMs有望增强自动驾驶系统的多个方面，从感知与场景理解到交互式决策。本文首先提出了设计用于自动驾驶的大语言模型（LLM4AD）这一新概念，随后回顾了现有LLM4AD研究。接着，本文提出了一个全面的基准测试体系，用于评估LLM4AD系统的指令遵循与推理能力，其中包括LaMPilot-Bench、仿真环境中的CARLA Leaderboard 1.0基准测试以及用于多视角视觉问答的NuPlanQA。此外，我们在真实自动驾驶车辆平台上开展了大量实验，考察了云端与边缘端LLM部署在个性化决策与运动控制中的表现。然后，探讨了将语言扩散模型集成至自动驾驶的未来趋势，并以所提出的ViLaD（视觉-语言扩散）框架为例加以说明。最后，讨论了LLM4AD面临的主要挑战，包括延迟、部署、安全与隐私、可靠性、信任与透明度以及个性化。