Multi-modal 3D object detection is important for reliable perception in robotics and autonomous driving. However, its effectiveness remains limited under adverse weather conditions due to weather-induced distortions and misalignment between different data modalities. In this work, we propose DiffFusion, a novel framework designed to enhance robustness in challenging weather through diffusion-based restoration and adaptive cross-modal fusion. Our key insight is that diffusion models possess strong capabilities for denoising and generating data that can adapt to various weather conditions. Building on this, DiffFusion introduces Diffusion-IR restoring images degraded by weather effects and Point Cloud Restoration (PCR) compensating for corrupted LiDAR data using image object cues. To tackle misalignments between two modalities, we develop Bidirectional Adaptive Fusion and Alignment Module (BAFAM). It enables dynamic multi-modal fusion and bidirectional bird's-eye view (BEV) alignment to maintain consistent spatial correspondence. Extensive experiments on three public datasets show that DiffFusion achieves state-of-the-art robustness under adverse weather while preserving strong clean-data performance. Zero-shot results on the real-world DENSE dataset further validate its generalization. The implementation of our DiffFusion will be released as open-source.

翻译：多模态三维目标检测对于机器人和自动驾驶中的可靠感知至关重要。然而，在恶劣天气条件下，由于天气引起的畸变以及不同数据模态之间的失准，其有效性仍然受限。在本研究中，我们提出了DiffFusion，一种新颖的框架，旨在通过基于扩散模型的恢复和自适应跨模态融合来增强在挑战性天气下的鲁棒性。我们的核心见解是，扩散模型具备强大的去噪和数据生成能力，能够适应各种天气条件。基于此，DiffFusion引入了Diffusion-IR来恢复受天气效应退化的图像，以及点云恢复（PCR）模块，利用图像目标线索补偿受损的LiDAR数据。为了解决两种模态之间的失准问题，我们开发了双向自适应融合与对齐模块（BAFAM）。该模块支持动态多模态融合和双向鸟瞰图（BEV）对齐，以保持一致的空间对应关系。在三个公开数据集上的大量实验表明，DiffFusion在恶劣天气下实现了最先进的鲁棒性，同时保持了强大的清洁数据性能。在真实世界DENSE数据集上的零样本结果进一步验证了其泛化能力。我们的DiffFusion实现将作为开源项目发布。