Despite significant recent progress in the field of autonomous driving, modern methods still struggle and can incur serious accidents when encountering long-tail unforeseen events and challenging urban scenarios. On the one hand, large language models (LLM) have shown impressive reasoning capabilities that approach "Artificial General Intelligence". On the other hand, previous autonomous driving methods tend to rely on limited-format inputs (e.g. sensor data and navigation waypoints), restricting the vehicle's ability to understand language information and interact with humans. To this end, this paper introduces LMDrive, a novel language-guided, end-to-end, closed-loop autonomous driving framework. LMDrive uniquely processes and integrates multi-modal sensor data with natural language instructions, enabling interaction with humans and navigation software in realistic instructional settings. To facilitate further research in language-based closed-loop autonomous driving, we also publicly release the corresponding dataset which includes approximately 64K instruction-following data clips, and the LangAuto benchmark that tests the system's ability to handle complex instructions and challenging driving scenarios. Extensive closed-loop experiments are conducted to demonstrate LMDrive's effectiveness. To the best of our knowledge, we're the very first work to leverage LLMs for closed-loop end-to-end autonomous driving. Codes, models, and datasets can be found at https://github.com/opendilab/LMDrive

翻译：尽管近年来自动驾驶领域取得了显著进展，但现有方法在处理长尾未预见事件和复杂城市场景时仍存在困难，甚至可能引发严重事故。一方面，大规模语言模型（LLM）展现出接近"通用人工智能"的卓越推理能力；另一方面，传统自动驾驶方法通常依赖有限格式输入（如传感器数据和导航路径点），限制了车辆理解语言信息及与人交互的能力。为此，本文提出LMDrive——一种新颖的语言驱动型闭环端到端自动驾驶框架。该框架独特地处理并融合多模态传感器数据与自然语言指令，使车辆能够在真实指令场景中与人类及导航软件交互。为促进基于语言的闭环自动驾驶研究，我们同步公开了包含约6.4万条指令跟随数据片段的数据集，以及用于测试系统处理复杂指令与挑战性驾驶场景能力的LangAuto基准。大量闭环实验证明了LMDrive的有效性。据我们所知，这是首个利用LLM实现闭环端到端自动驾驶的工作。代码、模型及数据集详见https://github.com/opendilab/LMDrive。