Recent advancements in autonomous driving have relied on data-driven approaches, which are widely adopted but face challenges including dataset bias, overfitting, and uninterpretability. Drawing inspiration from the knowledge-driven nature of human driving, we explore the question of how to instill similar capabilities into autonomous driving systems and summarize a paradigm that integrates an interactive environment, a driver agent, as well as a memory component to address this question. Leveraging large language models (LLMs) with emergent abilities, we propose the DiLu framework, which combines a Reasoning and a Reflection module to enable the system to perform decision-making based on common-sense knowledge and evolve continuously. Extensive experiments prove DiLu's capability to accumulate experience and demonstrate a significant advantage in generalization ability over reinforcement learning-based methods. Moreover, DiLu is able to directly acquire experiences from real-world datasets which highlights its potential to be deployed on practical autonomous driving systems. To the best of our knowledge, we are the first to leverage knowledge-driven capability in decision-making for autonomous vehicles. Through the proposed DiLu framework, LLM is strengthened to apply knowledge and to reason causally in the autonomous driving domain. Project page: https://pjlab-adg.github.io/DiLu/

翻译：近年来自动驾驶领域的进展主要依赖于数据驱动方法，这类方法虽被广泛采用，却面临数据集偏差、过拟合及不可解释性等挑战。受人类驾驶知识驱动特性的启发，我们探讨如何将类似能力注入自动驾驶系统，并归纳出一种集成交互环境、驾驶员智能体及记忆组件的范式。利用具有涌现能力的大语言模型，我们提出DiLu框架，该框架结合推理模块与反思模块，使系统能基于常识知识进行决策并持续进化。大量实验证明DiLu具备经验积累能力，且在泛化能力上显著优于基于强化学习的方法。此外，DiLu能直接从真实世界数据集获取经验，凸显其在实用自动驾驶系统中部署的潜力。据我们所知，这是首次在自动驾驶车辆决策中应用知识驱动能力。通过所提DiLu框架，大语言模型在自动驾驶领域被强化以应用知识并进行因果推理。项目页面：https://pjlab-adg.github.io/DiLu/