Despite significant recent progress in the field of autonomous driving, modern methods still struggle and can incur serious accidents when encountering long-tail unforeseen events and challenging urban scenarios. On the one hand, large language models (LLM) have shown impressive reasoning capabilities that approach "Artificial General Intelligence". On the other hand, previous autonomous driving methods tend to rely on limited-format inputs (e.g. sensor data and navigation waypoints), restricting the vehicle's ability to understand language information and interact with humans. To this end, this paper introduces LMDrive, a novel language-guided, end-to-end, closed-loop autonomous driving framework. LMDrive uniquely processes and integrates multi-modal sensor data with natural language instructions, enabling interaction with humans and navigation software in realistic instructional settings. To facilitate further research in language-based closed-loop autonomous driving, we also publicly release the corresponding dataset which includes approximately 64K instruction-following data clips, and the LangAuto benchmark that tests the system's ability to handle complex instructions and challenging driving scenarios. Extensive closed-loop experiments are conducted to demonstrate LMDrive's effectiveness. To the best of our knowledge, we're the very first work to leverage LLMs for closed-loop end-to-end autonomous driving. Codes can be found at https://github.com/opendilab/LMDrive

翻译：尽管近年来自动驾驶领域取得了显著进展，现有方法在面对长尾不可预见事件及复杂城市场景时仍易发生严重事故。一方面，大语言模型（LLM）展现出接近“通用人工智能”的惊人推理能力；另一方面，现有自动驾驶方法多依赖有限格式的输入（如传感器数据和导航航点），限制了车辆理解语言信息及与人类交互的能力。为此，本文提出LMDrive——一种新颖的语言引导式端到端闭环自动驾驶框架。LMDrive独特地融合多模态传感器数据与自然语言指令，在真实指令场景中实现与人类及导航软件的交互。为促进基于语言的闭环自动驾驶研究，我们同步公开了包含约6.4万条指令跟随数据片段的数据集，以及用于测试系统处理复杂指令与挑战性驾驶场景能力的LangAuto基准测试。大量闭环实验证明了LMDrive的有效性。据我们所知，这是首个利用大语言模型实现闭环端到端自动驾驶的工作。代码详见https://github.com/opendilab/LMDrive