In multi-modal learning, some modalities are more influential than others, and their absence can have a significant impact on classification/segmentation accuracy. Addressing this challenge, we propose a novel approach called Meta-learned Modality-weighted Knowledge Distillation (MetaKD), which enables multi-modal models to maintain high accuracy even when key modalities are missing. MetaKD adaptively estimates the importance weight of each modality through a meta-learning process. These learned importance weights guide a pairwise modality-weighted knowledge distillation process, allowing high-importance modalities to transfer knowledge to lower-importance ones, resulting in robust performance despite missing inputs. Unlike previous methods in the field, which are often task-specific and require significant modifications, our approach is designed to work in multiple tasks (e.g., segmentation and classification) with minimal adaptation. Experimental results on five prevalent datasets, including three Brain Tumor Segmentation datasets (BraTS2018, BraTS2019 and BraTS2020), the Alzheimer's Disease Neuroimaging Initiative (ADNI) classification dataset and the Audiovision-MNIST classification dataset, demonstrate the proposed model is able to outperform the compared models by a large margin.

翻译：在多模态学习中，某些模态比其他模态更具影响力，它们的缺失会对分类/分割精度产生显著影响。为应对这一挑战，我们提出了一种名为元学习模态加权知识蒸馏（MetaKD）的新方法，该方法使多模态模型即使在关键模态缺失时也能保持高精度。MetaKD通过元学习过程自适应地估计每个模态的重要性权重。这些学习到的重要性权重指导成对模态加权知识蒸馏过程，允许高重要性模态向低重要性模态传递知识，从而在输入缺失的情况下实现鲁棒性能。与领域内先前通常针对特定任务且需要重大修改的方法不同，我们的方法设计用于多种任务（例如分割和分类），且只需最小化适配。在五个主流数据集上的实验结果，包括三个脑肿瘤分割数据集（BraTS2018、BraTS2019和BraTS2020）、阿尔茨海默病神经影像学倡议（ADNI）分类数据集以及视听MNIST分类数据集，表明所提模型能够以较大优势超越对比模型。