Large Vision-Language Models (LVLMs) have received widespread attention in advancing the interpretable self-driving. Existing evaluations of LVLMs primarily focus on the multi-faceted capabilities in natural circumstances, lacking automated and quantifiable assessment for self-driving, let alone the severe road corner cases. In this paper, we propose CODA-LM, the very first benchmark for the automatic evaluation of LVLMs for self-driving corner cases. We adopt a hierarchical data structure to prompt powerful LVLMs to analyze complex driving scenes and generate high-quality pre-annotation for human annotators, and for LVLM evaluation, we show that using the text-only large language models (LLMs) as judges reveals even better alignment with human preferences than the LVLM judges. Moreover, with CODA-LM, we build CODA-VLM, a new driving LVLM surpassing all the open-sourced counterparts on CODA-LM. Our CODA-VLM performs comparably with GPT-4V, even surpassing GPT-4V by +21.42% on the regional perception task. We hope CODA-LM can become the catalyst to promote interpretable self-driving empowered by LVLMs.

翻译：大型视觉语言模型（LVLM）在推进可解释自动驾驶方面受到广泛关注。现有对LVLM的评估主要集中于其在自然场景下的多方面能力，缺乏针对自动驾驶的自动化、可量化评估，更遑论对严峻道路极端场景的评估。本文提出CODA-LM，首个针对自动驾驶极端场景的LVLM自动化评估基准。我们采用分层数据结构，提示强大的LVLM分析复杂驾驶场景并为人类标注者生成高质量预标注；对于LVLM评估，我们发现仅使用纯文本大语言模型（LLM）作为评判者，相比LVLM评判者能展现出与人类偏好更好的一致性。此外，基于CODA-LM，我们构建了CODA-VLM，这是一个新的驾驶专用LVLM，其在CODA-LM基准上超越了所有开源同类模型。我们的CODA-VLM性能与GPT-4V相当，在区域感知任务上甚至超越了GPT-4V达+21.42%。我们希望CODA-LM能成为催化剂，推动由LVLM赋能的、可解释的自动驾驶发展。