Robust 3D object detection in extreme weather and illumination conditions is a challenging task. While radars and thermal cameras are known for their resilience to these conditions, few studies have been conducted on radar-thermal fusion due to the lack of corresponding datasets. To address this gap, we first present a new multi-modal dataset called ThermRad, which includes a 3D LiDAR, a 4D radar, an RGB camera and a thermal camera. This dataset is unique because it includes data from all four sensors in extreme weather conditions, providing a valuable resource for future research in this area. To validate the robustness of 4D radars and thermal cameras for 3D object detection in challenging weather conditions, we propose a new multi-modal fusion method called RTDF-RCNN, which leverages the complementary strengths of 4D radars and thermal cameras to boost object detection performance. To further prove the effectiveness of our proposed framework, we re-implement state-of-the-art (SOTA) 3D detectors on our dataset as benchmarks for evaluation. Our method achieves significant enhancements in detecting cars, pedestrians, and cyclists, with improvements of over 7.98%, 24.27%, and 27.15%, respectively, while achieving comparable results to LiDAR-based approaches. Our contributions in both the ThermRad dataset and the new multi-modal fusion method provide a new approach to robust 3D object detection in adverse weather and illumination conditions. The ThermRad dataset will be released.

翻译：在极端天气和光照条件下实现鲁棒的三维目标检测是一项极具挑战性的任务。尽管雷达和热成像相机以其在恶劣环境下的适应性而闻名，但由于缺乏对应的数据集，关于雷达-热成像融合的研究仍十分有限。为填补这一空白，我们首先提出一个名为ThermRad的新型多模态数据集，该数据集包含3D激光雷达、4D雷达、RGB相机和热成像相机四种传感器。该数据集的独特之处在于，它涵盖了极端天气条件下所有四种传感器的数据，为这一领域的未来研究提供了宝贵资源。为验证4D雷达和热成像相机在恶劣天气下进行三维目标检测的鲁棒性，我们提出一种名为RTDF-RCNN的新型多模态融合方法，该方法利用4D雷达与热成像相机的互补优势来提升目标检测性能。为进一步证明所提框架的有效性，我们在数据集上重新实现了当前最先进（SOTA）的三维检测器作为评估基准。我们的方法在检测车辆、行人和自行车方面取得了显著提升，分别提高了7.98%、24.27%和27.15%，同时达到了与基于激光雷达方法相当的性能。我们在ThermRad数据集及新型多模态融合方法上的贡献，为恶劣天气和光照条件下的鲁棒三维目标检测提供了新方案。ThermRad数据集将公开发布。