Cross-modality images that integrate visible-infrared spectra cues can provide richer complementary information for object detection. Despite this, existing visible-infrared object detection methods severely degrade in severe weather conditions. This failure stems from the pronounced sensitivity of visible images to environmental perturbations, such as rain, haze, and snow, which frequently cause false negatives and false positives in detection. To address this issue, we introduce a novel and challenging task, termed visible-infrared object detection under adverse weather conditions. To foster this task, we have constructed a new Severe Weather Visible-Infrared Dataset (SWVID) with diverse severe weather scenes. Furthermore, we introduce the Cross-modality Fusion Mamba with Weather-removal (CFMW) to augment detection accuracy in adverse weather conditions. Thanks to the proposed Weather Removal Diffusion Model (WRDM) and Cross-modality Fusion Mamba (CFM) modules, CFMW is able to mine more essential information of pedestrian features in cross-modality fusion, thus could transfer to other rarer scenarios with high efficiency and has adequate availability on those platforms with low computing power. To the best of our knowledge, this is the first study that targeted improvement and integrated both Diffusion and Mamba modules in cross-modality object detection, successfully expanding the practical application of this type of model with its higher accuracy and more advanced architecture. Extensive experiments on both well-recognized and self-created datasets conclusively demonstrate that our CFMW achieves state-of-the-art detection performance, surpassing existing benchmarks. The dataset and source code will be made publicly available at https://github.com/lhy-zjut/CFMW.

翻译：跨模态图像融合了可见光和红外光谱线索，能够为目标检测提供更丰富的互补信息。然而，现有的可见光-红外目标检测方法在恶劣天气条件下性能严重退化。这种失效源于可见光图像对环境扰动（如雨、雾、雪）的显著敏感性，常导致检测中出现假阴性和假阳性。针对此问题，我们引入了一项新颖且具有挑战性的任务——恶劣天气条件下的可见光-红外目标检测。为促进该任务，我们构建了一个包含多种恶劣天气场景的新数据集——恶劣天气可见光-红外数据集（SWVID）。此外，我们提出了带有天气去除功能的跨模态融合Mamba（CFMW），以增强恶劣天气条件下的检测精度。得益于所提出的天气去除扩散模型（WRDM）和跨模态融合Mamba（CFM）模块，CFMW能够在跨模态融合中挖掘行人特征的更本质信息，从而高效迁移至其他更稀有场景，并在低算力平台上具有足够的可用性。据我们所知，这是首个在跨模态目标检测中针对性改进并整合扩散（Diffusion）和Mamba模块的研究，成功拓展了此类模型的实际应用，实现了更高的精度和更先进的架构。在公认数据及自建数据集上的大量实验充分证明，我们的CFMW达到了最先进的检测性能，超越了现有基准。该数据集和源代码将公开于https://github.com/lhy-zjut/CFMW。