Existing brain tumor segmentation methods usually utilize multiple Magnetic Resonance Imaging (MRI) modalities in brain tumor images for segmentation, which can achieve better segmentation performance. However, in clinical applications, some modalities are often missing due to resource constraints, resulting in significant performance degradation for methods that rely on complete modality segmentation. In this paper, we propose a Multimodal feature distillation with Mamba-Transformer hybrid network (MMTSeg) for accurate brain tumor segmentation with missing modalities. We first employ a Multimodal Feature Distillation (MFD) module to distill feature-level multimodal knowledge into different unimodalities to extract complete modality information. We further develop an Unimodal Feature Enhancement (UFE) module to model the semantic relationship between global and local information. Finally, we built a Cross-Modal Fusion (CMF) module to explicitly align the global correlations across modalities, even when some modalities are missing. Complementary features within and across modalities are refined by the Mamba-Transformer hybrid architectures in both the UFE and CMF modules, dynamically capturing long-range dependencies and global semantic information for complex spatial contexts. A boundary-wise loss function is employed as the segmentation loss of the proposed MMTSeg to minimize boundary discrepancies for a distance-based metric. Our ablation study demonstrates the importance of the proposed feature enhancement and fusion modules in the proposed network and the Transformer with Mamba block for improving the performance of brain tumor segmentation with missing modalities. Extensive experiments on the BraTS 2018 and BraTS 2020 datasets demonstrate that the proposed MMTSeg framework outperforms state-of-the-art methods when modalities are missing.

翻译：现有脑肿瘤分割方法通常利用脑肿瘤图像中的多种磁共振成像（MRI）模态进行分割，以获得更好的分割性能。然而，在临床应用中，由于资源限制，部分模态常常缺失，导致依赖完整模态的分割方法性能显著下降。本文提出了一种基于Mamba-Transformer混合网络的多模态特征蒸馏方法（MMTSeg），用于在模态缺失情况下实现精确的脑肿瘤分割。我们首先采用多模态特征蒸馏（MFD）模块，将特征级的多模态知识蒸馏到不同的单模态中，以提取完整的模态信息。进一步设计了单模态特征增强（UFE）模块，用于建模全局与局部信息之间的语义关系。最后，构建了跨模态融合（CMF）模块，以显式对齐跨模态的全局相关性，即使在某些模态缺失的情况下也能实现。在UFE和CMF模块中，通过Mamba-Transformer混合架构精炼模态内和跨模态的互补特征，动态捕获复杂空间上下文中的长程依赖和全局语义信息。采用边界感知损失函数作为所提MMTSeg的分割损失，以最小化基于距离度量的边界差异。消融实验证明了所提网络中特征增强与融合模块以及结合Mamba块的Transformer对于提升缺失模态下脑肿瘤分割性能的重要性。在BraTS 2018和BraTS 2020数据集上的大量实验表明，当模态缺失时，所提出的MMTSeg框架优于现有最先进方法。