Vision-Language Models (VLMs) and Multi-Modal Language models (MMLMs) have become prominent in autonomous driving research, as these models can provide interpretable textual reasoning and responses for end-to-end autonomous driving safety tasks using traffic scene images and other data modalities. However, current approaches to these systems use expensive large language model (LLM) backbones and image encoders, making such systems unsuitable for real-time autonomous driving systems where tight memory constraints exist and fast inference time is necessary. To address these previous issues, we develop EM-VLM4AD, an efficient, lightweight, multi-frame vision language model which performs Visual Question Answering for autonomous driving. In comparison to previous approaches, EM-VLM4AD requires at least 10 times less memory and floating point operations, while also achieving higher CIDEr and ROUGE-L scores than the existing baseline on the DriveLM dataset. EM-VLM4AD also exhibits the ability to extract relevant information from traffic views related to prompts and can answer questions for various autonomous driving subtasks. We release our code to train and evaluate our model at https://github.com/akshaygopalkr/EM-VLM4AD.

翻译：视觉语言模型（VLM）和多模态语言模型（MMLM）已在自动驾驶研究中占据重要地位，这些模型可利用交通场景图像及其他数据模态，为端到端自动驾驶安全任务提供可解释的文本推理与响应。然而，当前此类系统采用昂贵的大语言模型（LLM）骨干网络与图像编码器，导致其难以适用于存在严格内存限制且需快速推理的实时自动驾驶系统。为解决上述问题，我们提出了EM-VLM4AD——一种专为自动驾驶视觉问答任务设计的高效轻量多帧视觉语言模型。与现有方法相比，EM-VLM4AD至少减少了10倍的内存占用与浮点运算量，同时在DriveLM数据集上取得了优于现有基线的CIDEr与ROUGE-L评分。此外，该模型能够从与提示相关的交通视角中提取有效信息，并回答各类自动驾驶子任务问题。我们在https://github.com/akshaygopalkr/EM-VLM4AD上发布了模型训练与评估代码。