Large language models (LLMs) have opened up new possibilities for intelligent agents, endowing them with human-like thinking and cognitive abilities. In this work, we delve into the potential of large language models (LLMs) in autonomous driving (AD). We introduce DriveMLM, an LLM-based AD framework that can perform close-loop autonomous driving in realistic simulators. To this end, (1) we bridge the gap between the language decisions and the vehicle control commands by standardizing the decision states according to the off-the-shelf motion planning module. (2) We employ a multi-modal LLM (MLLM) to model the behavior planning module of a module AD system, which uses driving rules, user commands, and inputs from various sensors (e.g., camera, lidar) as input and makes driving decisions and provide explanations; This model can plug-and-play in existing AD systems such as Apollo for close-loop driving. (3) We design an effective data engine to collect a dataset that includes decision state and corresponding explanation annotation for model training and evaluation. We conduct extensive experiments and show that our model achieves 76.1 driving score on the CARLA Town05 Long, and surpasses the Apollo baseline by 4.7 points under the same settings, demonstrating the effectiveness of our model. We hope this work can serve as a baseline for autonomous driving with LLMs. Code and models shall be released at https://github.com/OpenGVLab/DriveMLM.

翻译：大型语言模型（LLM）为智能体开辟了新的可能性，赋予其类人思维与认知能力。本文深入探究了大型语言模型在自动驾驶领域的潜力。我们提出了DriveMLM——一种基于LLM的自动驾驶框架，能够在真实模拟器中执行闭环自动驾驶。为此：（1）我们通过根据现成的运动规划模块标准化决策状态，弥合了语言决策与车辆控制指令之间的鸿沟；（2）采用多模态大语言模型（MLLM）对模块化自动驾驶系统中的行为规划模块进行建模，该模型以驾驶规则、用户指令及各类传感器（如摄像头、激光雷达）输入作为依据，做出驾驶决策并提供解释说明；该模型可即插即用地集成至现有自动驾驶系统（如Apollo）中实现闭环驾驶；（3）我们设计了一套高效的数据引擎，收集包含决策状态及相应解释标注的数据集，用于模型训练与评估。通过大量实验表明，我们的模型在CARLA Town05 Long场景中取得了76.1的驾驶得分，并在相同设置下以4.7分的优势超越Apollo基线，验证了其有效性。我们期望该项工作能为基于LLM的自动驾驶研究提供基线参考。代码与模型将发布于https://github.com/OpenGVLab/DriveMLM。