Integrating large language models (LLMs) in autonomous vehicles enables conversation with AI systems to drive the vehicle. However, it also emphasizes the requirement for such systems to comprehend commands accurately and achieve higher-level personalization to adapt to the preferences of drivers or passengers over a more extended period. In this paper, we introduce an LLM-based framework, Talk2Drive, capable of translating natural verbal commands into executable controls and learning to satisfy personal preferences for safety, efficiency, and comfort with a proposed memory module. This is the first-of-its-kind multi-scenario field experiment that deploys LLMs on a real-world autonomous vehicle. Experiments showcase that the proposed system can comprehend human intentions at different intuition levels, ranging from direct commands like "can you drive faster" to indirect commands like "I am really in a hurry now". Additionally, we use the takeover rate to quantify the trust of human drivers in the LLM-based autonomous driving system, where Talk2Drive significantly reduces the takeover rate in highway, intersection, and parking scenarios. We also validate that the proposed memory module considers personalized preferences and further reduces the takeover rate by up to 65.2% compared with those without a memory module. The experiment video can be watched at https://www.youtube.com/watch?v=4BWsfPaq1Ro

翻译：将大语言模型（LLMs）集成到自动驾驶车辆中，使得通过与人工智能系统对话来驾驶车辆成为可能。然而，这也强调了此类系统需要准确理解指令，并实现更高水平的个性化，以在更长时期内适应驾驶员或乘客的偏好。在本文中，我们提出了一种基于LLM的框架Talk2Drive，该框架能够将自然语言口头指令转化为可执行的控制操作，并通过所提出的记忆模块学习满足个人在安全性、效率和舒适性方面的偏好。这是首个在真实自动驾驶车辆上部署LLMs的多场景实地实验。实验表明，所提出的系统能够理解不同直觉层面的人类意图，从“你能开快点吗”这样的直接指令到“我现在真的很着急”这样的间接指令。此外，我们使用接管率来量化人类驾驶员对基于LLM的自动驾驶系统的信任度，其中Talk2Drive在高速公路、交叉路口和停车场场景中显著降低了接管率。我们还验证了所提出的记忆模块考虑了个性化偏好，与没有记忆模块的系统相比，进一步将接管率降低了高达65.2%。实验视频可访问https://www.youtube.com/watch?v=4BWsfPaq1Ro观看。