Achieving level-5 driving automation in autonomous vehicles necessitates a robust semantic visual perception system capable of parsing data from different sensors across diverse conditions. However, existing semantic perception datasets often lack important non-camera modalities typically used in autonomous vehicles, or they do not exploit such modalities to aid and improve semantic annotations in challenging conditions. To address this, we introduce MUSES, the MUlti-SEnsor Semantic perception dataset for driving in adverse conditions under increased uncertainty. MUSES includes synchronized multimodal recordings with 2D panoptic annotations for 2500 images captured under diverse weather and illumination. The dataset integrates a frame camera, a lidar, a radar, an event camera, and an IMU/GNSS sensor. Our new two-stage panoptic annotation protocol captures both class-level and instance-level uncertainty in the ground truth and enables the novel task of uncertainty-aware panoptic segmentation we introduce, along with standard semantic and panoptic segmentation. MUSES proves both effective for training and challenging for evaluating models under diverse visual conditions, and it opens new avenues for research in multimodal and uncertainty-aware dense semantic perception. Our dataset and benchmark will be made publicly available.

翻译：实现自动驾驶车辆的L5级全自动驾驶，需要鲁棒的语义视觉感知系统，能够解析来自不同传感器在多种条件下的数据。然而，现有语义感知数据集往往缺乏自动驾驶车辆中重要的非摄像头模态，或未利用此类模态辅助改进恶劣条件下的语义标注。为此，我们提出MUSES——多传感器不确定驾驶条件下的语义感知数据集。MUSES包含2500张多样化天气和光照条件下同步采集的多模态影像及二维全景标注。数据集整合了帧相机、激光雷达、毫米波雷达、事件相机及惯性测量单元/全球导航卫星系统传感器。我们提出的新型两阶段全景标注协议可捕获真实场景中的类别级和实例级不确定性，并支持我们引入的不确定性感知全景分割新任务及标准语义与全景分割任务。MUSES既证明其在多样化视觉条件下训练模型的有效性，又保持对模型评估的挑战性，为多模态及不确定性感知密集语义感知研究开辟新方向。本数据集及基准测试将公开开放。