Multimodal magnetic resonance imaging (MRI) provides complementary information for sub-region analysis of brain tumors. Plenty of methods have been proposed for automatic brain tumor segmentation using four common MRI modalities and achieved remarkable performance. In practice, however, it is common to have one or more modalities missing due to image corruption, artifacts, acquisition protocols, allergy to contrast agents, or simply cost. In this work, we propose a novel two-stage framework for brain tumor segmentation with missing modalities. In the first stage, a multimodal masked autoencoder (M3AE) is proposed, where both random modalities (i.e., modality dropout) and random patches of the remaining modalities are masked for a reconstruction task, for self-supervised learning of robust multimodal representations against missing modalities. To this end, we name our framework M3AE. Meanwhile, we employ model inversion to optimize a representative full-modal image at marginal extra cost, which will be used to substitute for the missing modalities and boost performance during inference. Then in the second stage, a memory-efficient self distillation is proposed to distill knowledge between heterogenous missing-modal situations while fine-tuning the model for supervised segmentation. Our M3AE belongs to the 'catch-all' genre where a single model can be applied to all possible subsets of modalities, thus is economic for both training and deployment. Extensive experiments on BraTS 2018 and 2020 datasets demonstrate its superior performance to existing state-of-the-art methods with missing modalities, as well as the efficacy of its components. Our code is available at: https://github.com/ccarliu/m3ae.

翻译：多模态磁共振成像（MRI）为脑肿瘤子区域分析提供了互补信息。现有大量方法利用四种常见MRI模态实现自动脑肿瘤分割，并取得了显著性能。然而实际应用中，由于图像损坏、伪影、采集协议、造影剂过敏或单纯成本问题，常出现一种或多种模态缺失的情况。本文提出一种新颖的两阶段框架，用于处理缺失模态下的脑肿瘤分割任务。第一阶段提出多模态掩码自编码器（M3AE），通过对随机模态（即模态丢弃）和剩余模态的随机图像块进行掩码重建，实现针对缺失模态的鲁棒多模态表示自监督学习。为此，我们将该框架命名为M3AE。同时，我们采用模型反演以极低的额外成本优化代表性全模态图像，用于在推理阶段替代缺失模态并提升性能。第二阶段提出内存高效的自蒸馏方法，在微调模型进行监督分割的同时，实现异质缺失模态场景间的知识蒸馏。我们的M3AE属于"全能型"方法，单个模型可应用于所有可能的模态子集组合，因此训练和部署成本均较低。在BraTS 2018和2020数据集上的大量实验表明，该方法在缺失模态场景下优于现有最优方法，且各组件均具有效性。代码开源地址：https://github.com/ccarliu/m3ae。