We introduce EMMA, an End-to-end Multimodal Model for Autonomous driving. Built on a multi-modal large language model foundation, EMMA directly maps raw camera sensor data into various driving-specific outputs, including planner trajectories, perception objects, and road graph elements. EMMA maximizes the utility of world knowledge from the pre-trained large language models, by representing all non-sensor inputs (e.g. navigation instructions and ego vehicle status) and outputs (e.g. trajectories and 3D locations) as natural language text. This approach allows EMMA to jointly process various driving tasks in a unified language space, and generate the outputs for each task using task-specific prompts. Empirically, we demonstrate EMMA's effectiveness by achieving state-of-the-art performance in motion planning on nuScenes as well as competitive results on the Waymo Open Motion Dataset (WOMD). EMMA also yields competitive results for camera-primary 3D object detection on the Waymo Open Dataset (WOD). We show that co-training EMMA with planner trajectories, object detection, and road graph tasks yields improvements across all three domains, highlighting EMMA's potential as a generalist model for autonomous driving applications. However, EMMA also exhibits certain limitations: it can process only a small amount of image frames, does not incorporate accurate 3D sensing modalities like LiDAR or radar and is computationally expensive. We hope that our results will inspire further research to mitigate these issues and to further evolve the state of the art in autonomous driving model architectures.

翻译：本文提出EMMA，一种面向自动驾驶的端到端多模态模型。该模型基于多模态大语言模型架构，能够直接将原始摄像头传感器数据映射为多种驾驶专用输出，包括规划轨迹、感知物体及道路图元素。EMMA通过将所有非传感器输入（如导航指令与自车状态）和输出（如轨迹与3D位置）表示为自然语言文本，最大限度地利用了预训练大语言模型中的世界知识。该方法使EMMA能够在统一语言空间中协同处理多种驾驶任务，并通过任务特定提示生成各任务输出。实验表明，EMMA在nuScenes数据集上的运动规划任务中取得了最先进的性能，在Waymo开放运动数据集（WOMD）上也获得了具有竞争力的结果。同时，EMMA在Waymo开放数据集（WOD）的摄像头主导3D物体检测任务中表现优异。我们证明，通过联合训练规划轨迹、物体检测和道路图任务，EMMA在所有三个领域均获得性能提升，彰显了其作为自动驾驶通用模型的潜力。然而，EMMA也存在一定局限性：仅能处理少量图像帧，未集成LiDAR或雷达等精确3D传感模态，且计算成本较高。我们希望本研究能启发后续工作克服这些限制，推动自动驾驶模型架构的进一步发展。