Building a multi-modality multi-task neural network toward accurate and robust performance is a de-facto standard in perception task of autonomous driving. However, leveraging such data from multiple sensors to jointly optimize the prediction and planning tasks remains largely unexplored. In this paper, we present FusionAD, to the best of our knowledge, the first unified framework that fuse the information from two most critical sensors, camera and LiDAR, goes beyond perception task. Concretely, we first build a transformer based multi-modality fusion network to effectively produce fusion based features. In constrast to camera-based end-to-end method UniAD, we then establish a fusion aided modality-aware prediction and status-aware planning modules, dubbed FMSPnP that take advantages of multi-modality features. We conduct extensive experiments on commonly used benchmark nuScenes dataset, our FusionAD achieves state-of-the-art performance and surpassing baselines on average 15% on perception tasks like detection and tracking, 10% on occupancy prediction accuracy, reducing prediction error from 0.708 to 0.389 in ADE score and reduces the collision rate from 0.31% to only 0.12%.

翻译：构建一个具备精确性和鲁棒性的多模态多任务神经网络，是自动驾驶感知任务的行业标准。然而，如何利用来自多个传感器的数据联合优化预测与规划任务，至今仍鲜有研究。本文提出了FusionAD——据我们所知，首个将两大关键传感器（摄像头和激光雷达）的信息融合并超越感知任务的统一框架。具体而言，我们首先构建了一个基于Transformer的多模态融合网络，以高效生成基于融合的特征。与基于摄像头的端到端方法UniAD不同，我们随后建立了融合辅助的模态感知预测模块和状态感知规划模块（称为FMSPnP），充分利用多模态特征。我们在通用基准数据集nuScenes上进行了大量实验，结果表明：FusionAD实现了最先进的性能，在检测和跟踪等感知任务上平均超越基线15%，在占用率预测精度上提升10%，将预测误差ADE得分从0.708降至0.389，并将碰撞率从0.31%降至仅0.12%。