Robust 3D object detection in extreme weather and illumination conditions is a challenging task. While radars and thermal cameras are known for their resilience to these conditions, few studies have been conducted on radar-thermal fusion due to the lack of corresponding datasets. To address this gap, we first present a new multi-modal dataset called ThermRad, which includes a 3D LiDAR, a 4D radar, an RGB camera and a thermal camera. This dataset is unique because it includes data from all four sensors in extreme weather conditions, providing a valuable resource for future research in this area. To validate the robustness of 4D radars and thermal cameras for 3D object detection in challenging weather conditions, we propose a new multi-modal fusion method called RTDF-RCNN, which leverages the complementary strengths of 4D radars and thermal cameras to boost object detection performance. To further prove the effectiveness of our proposed framework, we re-implement state-of-the-art (SOTA) 3D detectors on our dataset as benchmarks for evaluation. Our method achieves significant enhancements in detecting cars, pedestrians, and cyclists, with improvements of over 7.98%, 24.27%, and 27.15%, respectively, while achieving comparable results to LiDAR-based approaches. Our contributions in both the ThermRad dataset and the new multi-modal fusion method provide a new approach to robust 3D object detection in adverse weather and illumination conditions. The ThermRad dataset will be released.

翻译：在极端天气与光照条件下实现鲁棒的三维目标检测是一项具有挑战性的任务。尽管雷达和热成像相机以其对恶劣条件的适应性著称，但由于缺乏相应的数据集，目前针对雷达-热成像融合的研究较少。为弥补这一空白，我们首先提出一个名为ThermRad的新型多模态数据集，该数据集包含三维激光雷达、四维雷达、可见光相机和热成像相机四类传感器。该数据集的独特性在于它涵盖了极端天气条件下所有四种传感器的数据，为该领域的未来研究提供了宝贵资源。为验证四维雷达与热成像相机在恶劣天气条件下进行三维目标检测的鲁棒性，我们提出一种名为RTDF-RCNN的多模态融合方法，该方法利用四维雷达与热成像相机的互补优势以提升目标检测性能。为进一步证明所提框架的有效性，我们在数据集上复现了当前最优的三维检测器作为评估基准。我们的方法在检测车辆、行人和骑行者方面分别实现了超过7.98%、24.27%和27.15%的显著性能提升，并达到了与基于激光雷达的方法相当的结果。我们在ThermRad数据集及新型多模态融合方法上的贡献，为恶劣天气与光照条件下的鲁棒三维目标检测提供了新思路。ThermRad数据集将公开释放。