Multi-modal 3D object detection is important for reliable perception in robotics and autonomous driving. However, its effectiveness remains limited under adverse weather conditions due to weather-induced distortions and misalignment between different data modalities. In this work, we propose DiffFusion, a novel framework designed to enhance robustness in challenging weather through diffusion-based restoration and adaptive cross-modal fusion. Our key insight is that diffusion models possess strong capabilities for denoising and generating data that can adapt to various weather conditions. Building on this, DiffFusion introduces Diffusion-IR restoring images degraded by weather effects and Point Cloud Restoration (PCR) compensating for corrupted LiDAR data using image object cues. To tackle misalignments between two modalities, we develop Bidirectional Adaptive Fusion and Alignment Module (BAFAM). It enables dynamic multi-modal fusion and bidirectional bird's-eye view (BEV) alignment to maintain consistent spatial correspondence. Extensive experiments on three public datasets show that DiffFusion achieves state-of-the-art robustness under adverse weather while preserving strong clean-data performance. Zero-shot results on the real-world DENSE dataset further validate its generalization. The implementation of our DiffFusion will be released as open-source.

翻译：多模态三维目标检测对于机器人与自动驾驶的可靠感知至关重要。然而，在恶劣天气条件下，由于天气引起的畸变以及不同数据模态之间的错位，其有效性仍受到限制。本研究提出DiffFusion，一种基于扩散模型恢复与自适应跨模态融合的新型框架，旨在提升系统在挑战性天气条件下的鲁棒性。我们的核心观点在于，扩散模型具备强大的去噪与数据生成能力，能够适应多种天气条件。基于此，DiffFusion引入了Diffusion-IR模块以恢复受天气效应退化的图像，以及点云恢复模块，利用图像目标线索补偿受损的LiDAR数据。为解决双模态间的错位问题，我们开发了双向自适应融合与对齐模块，该模块支持动态多模态融合及双向鸟瞰图对齐，以保持空间对应关系的一致性。在三个公开数据集上的大量实验表明，DiffFusion在恶劣天气下实现了最先进的鲁棒性，同时保持了优异的清洁数据性能。在真实世界DENSE数据集上的零样本结果进一步验证了其泛化能力。DiffFusion的实现将作为开源项目发布。