Modern multispectral feature fusion for object detection faces two critical limitations: (1) Excessive preference for local complementary features over cross-modal shared semantics adversely affects generalization performance; and (2) The trade-off between the receptive field size and computational complexity present critical bottlenecks for scalable feature modeling. Addressing these issues, a novel Multispectral State-Space Feature Fusion framework, dubbed MS2Fusion, is proposed based on the state space model (SSM), achieving efficient and effective fusion through a dual-path parametric interaction mechanism. More specifically, the first cross-parameter interaction branch inherits the advantage of cross-attention in mining complementary information with cross-modal hidden state decoding in SSM. The second shared-parameter branch explores cross-modal alignment with joint embedding to obtain cross-modal similar semantic features and structures through parameter sharing in SSM. Finally, these two paths are jointly optimized with SSM for fusing multispectral features in a unified framework, allowing our MS2Fusion to enjoy both functional complementarity and shared semantic space. In our extensive experiments on mainstream benchmarks including FLIR, M3FD and LLVIP, our MS2Fusion significantly outperforms other state-of-the-art multispectral object detection methods, evidencing its superiority. Moreover, MS2Fusion is general and applicable to other multispectral perception tasks. We show that, even without specific design, MS2Fusion achieves state-of-the-art results on RGB-T semantic segmentation and RGBT salient object detection, showing its generality. The source code will be available at https://github.com/61s61min/MS2Fusion.git.

翻译：现代多光谱特征融合在目标检测中面临两个关键局限：（1）过度偏好局部互补特征而忽视跨模态共享语义，这不利于泛化性能；（2）感受野大小与计算复杂度之间的权衡成为可扩展特征建模的关键瓶颈。针对这些问题，本文基于状态空间模型（SSM）提出了一种新颖的多光谱状态空间特征融合框架，命名为MS2Fusion，通过双路径参数交互机制实现高效且有效的融合。具体而言，第一个跨参数交互分支继承了交叉注意力在挖掘互补信息方面的优势，并结合SSM中的跨模态隐藏状态解码；第二个共享参数分支通过SSM中的参数共享探索跨模态对齐与联合嵌入，以获取跨模态相似语义特征和结构。最终，这两条路径在SSM中联合优化，在统一框架下融合多光谱特征，使MS2Fusion兼具功能互补性和共享语义空间。在包括FLIR、M3FD和LLVIP在内的主流基准测试中，我们进行了广泛实验，结果表明MS2Fusion显著优于其他最先进的多光谱目标检测方法，证明了其优越性。此外，MS2Fusion具有通用性，可应用于其他多光谱感知任务。我们证明，即使未经专门设计，MS2Fusion在RGB-T语义分割和RGBT显著目标检测任务上也达到了最先进的性能，展现了其普适性。源代码将在https://github.com/61s61min/MS2Fusion.git公开。