Perception systems in modern autonomous driving vehicles typically take inputs from complementary multi-modal sensors, e.g., LiDAR and cameras. However, in real-world applications, sensor corruptions and failures lead to inferior performances, thus compromising autonomous safety. In this paper, we propose a robust framework, called MetaBEV, to address extreme real-world environments involving overall six sensor corruptions and two extreme sensor-missing situations. In MetaBEV, signals from multiple sensors are first processed by modal-specific encoders. Subsequently, a set of dense BEV queries are initialized, termed meta-BEV. These queries are then processed iteratively by a BEV-Evolving decoder, which selectively aggregates deep features from either LiDAR, cameras, or both modalities. The updated BEV representations are further leveraged for multiple 3D prediction tasks. Additionally, we introduce a new M2oE structure to alleviate the performance drop on distinct tasks in multi-task joint learning. Finally, MetaBEV is evaluated on the nuScenes dataset with 3D object detection and BEV map segmentation tasks. Experiments show MetaBEV outperforms prior arts by a large margin on both full and corrupted modalities. For instance, when the LiDAR signal is missing, MetaBEV improves 35.5% detection NDS and 17.7% segmentation mIoU upon the vanilla BEVFusion model; and when the camera signal is absent, MetaBEV still achieves 69.2% NDS and 53.7% mIoU, which is even higher than previous works that perform on full-modalities. Moreover, MetaBEV performs fairly against previous methods in both canonical perception and multi-task learning settings, refreshing state-of-the-art nuScenes BEV map segmentation with 70.4% mIoU.

翻译：现代自动驾驶车辆的感知系统通常依赖互补的多模态传感器（例如激光雷达和摄像头）进行输入。然而，在现实应用中，传感器损坏和故障会导致性能下降，从而危及自动驾驶安全。本文提出一种鲁棒性框架MetaBEV，以应对极端现实环境，包括六种传感器损坏和两种极端传感器缺失情况。在MetaBEV中，来自多传感器的信号首先通过模态特定编码器处理。随后，初始化一组密集的BEV查询，称为Meta-BEV。这些查询通过BEV-Evolving解码器迭代处理，该解码器选择性地从激光雷达、摄像头或两者模态中聚合深层特征。更新后的BEV表示进一步用于多项3D预测任务。此外，我们引入新的M2oE结构，以缓解多任务联合学习中不同任务上的性能下降。最终，MetaBEV在nuScenes数据集上结合3D目标检测和BEV地图分割任务进行评估。实验表明，在完整和损坏模态下，MetaBEV均大幅超越先前方法。例如，当激光雷达信号缺失时，MetaBEV在原始BEVFusion模型基础上将检测NDS提升35.5%，分割mIoU提升17.7%；当摄像头信号缺失时，MetaBEV仍达到69.2% NDS和53.7% mIoU，甚至高于先前在全模态下的工作性能。此外，MetaBEV在标准感知和多任务学习设定下与先前方法性能相当，并在nuScenes BEV地图分割中以70.4% mIoU刷新了最先进结果。