Multi-modality (MM) image fusion aims to render fused images that maintain the merits of different modalities, e.g., functional highlight and detailed textures. To tackle the challenge in modeling cross-modality features and decomposing desirable modality-specific and modality-shared features, we propose a novel Correlation-Driven feature Decomposition Fusion (CDDFuse) network. Firstly, CDDFuse uses Restormer blocks to extract cross-modality shallow features. We then introduce a dual-branch Transformer-CNN feature extractor with Lite Transformer (LT) blocks leveraging long-range attention to handle low-frequency global features and Invertible Neural Networks (INN) blocks focusing on extracting high-frequency local information. A correlation-driven loss is further proposed to make the low-frequency features correlated while the high-frequency features uncorrelated based on the embedded information. Then, the LT-based global fusion and INN-based local fusion layers output the fused image. Extensive experiments demonstrate that our CDDFuse achieves promising results in multiple fusion tasks, including infrared-visible image fusion and medical image fusion. We also show that CDDFuse can boost the performance in downstream infrared-visible semantic segmentation and object detection in a unified benchmark. The code is available at https://github.com/Zhaozixiang1228/MMIF-CDDFuse.

翻译：多模态图像融合旨在生成保留不同模态优势（如功能高亮与细节纹理）的融合图像。为应对跨模态特征建模及理想模态专属与模态共享特征分解的挑战，我们提出一种新颖的相关性驱动特征分解融合网络（CDDFuse）。首先，CDDFuse 采用Restormer模块提取跨模态浅层特征；随后引入双分支Transformer-CNN特征提取器，其中轻量Transformer（LT）模块通过长程注意力处理低频全局特征，可逆神经网络（INN）模块聚焦于高频局部信息提取。基于嵌入信息，进一步提出相关性损失函数，使低频特征保持相关而高频特征不相关。继而，基于LT的全局融合层与基于INN的局部融合层输出融合图像。大量实验表明，CDDFuse 在红外-可见光图像融合及医学图像融合等多个融合任务中取得优异效果。我们还证明，CDDFuse 可在统一基准下提升下游红外-可见光语义分割与目标检测性能。代码开源见：https://github.com/Zhaozixiang1228/MMIF-CDDFuse。