Achieving level-5 driving automation in autonomous vehicles necessitates a robust semantic visual perception system capable of parsing data from different sensors across diverse conditions. However, existing semantic perception datasets often lack important non-camera modalities typically used in autonomous vehicles, or they do not exploit such modalities to aid and improve semantic annotations in challenging conditions. To address this, we introduce MUSES, the MUlti-SEnsor Semantic perception dataset for driving in adverse conditions under increased uncertainty. MUSES includes synchronized multimodal recordings with 2D panoptic annotations for 2500 images captured under diverse weather and illumination. The dataset integrates a frame camera, a lidar, a radar, an event camera, and an IMU/GNSS sensor. Our new two-stage panoptic annotation protocol captures both class-level and instance-level uncertainty in the ground truth and enables the novel task of uncertainty-aware panoptic segmentation we introduce, along with standard semantic and panoptic segmentation. MUSES proves both effective for training and challenging for evaluating models under diverse visual conditions, and it opens new avenues for research in multimodal and uncertainty-aware dense semantic perception. Our dataset and benchmark are publicly available at https://muses.vision.ee.ethz.ch.

翻译：实现自动驾驶车辆的五级自动化需要一个鲁棒的语义视觉感知系统，能够解析不同条件下来自多种传感器的数据。然而，现有的语义感知数据集通常缺乏自动驾驶车辆中常用的重要非相机模态，或者未能利用这些模态来辅助和改进在挑战性条件下的语义标注。为此，我们提出了MUSES，这是一个针对不确定性增强的恶劣驾驶环境的多传感器语义感知数据集。MUSES包含了2500张在不同天气和光照条件下捕获的图像，提供了同步的多模态数据记录及二维全景标注。该数据集集成了帧相机、激光雷达、雷达、事件相机以及IMU/GNSS传感器。我们提出的新型两阶段全景标注协议能够捕捉真值中的类别级和实例级不确定性，并支持我们引入的新任务——不确定性感知的全景分割，以及标准的语义分割和全景分割。MUSES被证明在多样视觉条件下既能有效用于模型训练，也对模型评估构成挑战，并为多模态及不确定性感知的密集语义感知研究开辟了新途径。我们的数据集与基准测试已在https://muses.vision.ee.ethz.ch公开提供。