Cooperative perception offers several benefits for enhancing the capabilities of autonomous vehicles and improving road safety. Using roadside sensors in addition to onboard sensors increases reliability and extends the sensor range. External sensors offer higher situational awareness for automated vehicles and prevent occlusions. We propose CoopDet3D, a cooperative multi-modal fusion model, and TUMTraf-V2X, a perception dataset, for the cooperative 3D object detection and tracking task. Our dataset contains 2,000 labeled point clouds and 5,000 labeled images from five roadside and four onboard sensors. It includes 30k 3D boxes with track IDs and precise GPS and IMU data. We labeled eight categories and covered occlusion scenarios with challenging driving maneuvers, like traffic violations, near-miss events, overtaking, and U-turns. Through multiple experiments, we show that our CoopDet3D camera-LiDAR fusion model achieves an increase of +14.36 3D mAP compared to a vehicle camera-LiDAR fusion model. Finally, we make our dataset, model, labeling tool, and dev-kit publicly available on our website: https://tum-traffic-dataset.github.io/tumtraf-v2x.

翻译：协同感知在增强自动驾驶车辆能力及提升道路安全方面具有诸多优势。在车载传感器基础上增设路侧传感器，可提升系统可靠性并扩展传感器感知范围。外部传感器为自动驾驶车辆提供更高情境感知能力，并能有效避免遮挡问题。我们提出了面向协同3D目标检测与跟踪任务的CoopDet3D协同多模态融合模型及TUMTraf-V2X感知数据集。该数据集包含来自五个路侧传感器和四个车载传感器的2000组标注点云与5000张标注图像，涵盖30k个带轨迹ID的3D边界框及精密的GPS和IMU数据。我们标注了八个类别，并覆盖了含挑战性驾驶操作（如交通违规、险情事件、超车及掉头）的遮挡场景。通过多组实验表明，相比车载相机-激光雷达融合模型，我们提出的CoopDet3D相机-激光雷达融合模型在3D mAP指标上提升了+14.36。最后，我们已在官网https://tum-traffic-dataset.github.io/tumtraf-v2x开源了该数据集、模型、标注工具及开发工具包。