The use of multimodal data in assisted diagnosis and segmentation has emerged as a prominent area of interest in current research. However, one of the primary challenges is how to effectively fuse multimodal features. Most of the current approaches focus on the integration of multimodal features while ignoring the correlation and consistency between different modal features, leading to the inclusion of potentially irrelevant information. To address this issue, we introduce an innovative Multimodal Information Cross Transformer (MicFormer), which employs a dual-stream architecture to simultaneously extract features from each modality. Leveraging the Cross Transformer, it queries features from one modality and retrieves corresponding responses from another, facilitating effective communication between bimodal features. Additionally, we incorporate a deformable Transformer architecture to expand the search space. We conducted experiments on the MM-WHS dataset, and in the CT-MRI multimodal image segmentation task, we successfully improved the whole-heart segmentation DICE score to 85.57 and MIoU to 75.51. Compared to other multimodal segmentation techniques, our method outperforms by margins of 2.83 and 4.23, respectively. This demonstrates the efficacy of MicFormer in integrating relevant information between different modalities in multimodal tasks. These findings hold significant implications for multimodal image tasks, and we believe that MicFormer possesses extensive potential for broader applications across various domains. Access to our method is available at https://github.com/fxxJuses/MICFormer

翻译：多模态数据在辅助诊断与分割中的应用已成为当前研究的热点领域。然而，主要挑战之一在于如何有效融合多模态特征。当前大多数方法侧重于多模态特征的整合，却忽视了不同模态特征之间的相关性与一致性，导致可能引入无关信息。为解决这一问题，我们提出了一种创新的多模态信息交叉Transformer（MicFormer），该模型采用双流架构同时提取各模态特征。通过利用交叉Transformer，模型从一种模态查询特征，并从另一种模态检索对应响应，从而促进双模态特征间的有效交互。此外，我们引入了可变形Transformer架构以扩展搜索空间。在MM-WHS数据集上进行的实验中，我们在CT-MRI多模态图像分割任务中成功将全心分割DICE分数提升至85.57，MIoU提升至75.51。与其他多模态分割技术相比，我们的方法分别领先2.83和4.23个百分点。这证明了MicFormer在多模态任务中整合不同模态相关信息的有效性。这些发现对多模态图像任务具有重要启示意义，我们相信MicFormer在各领域具有广泛的应用潜力。方法代码可访问https://github.com/fxxJuses/MICFormer获取。