The missing modality issue is critical but non-trivial to be solved by multi-modal models. Current methods aiming to handle the missing modality problem in multi-modal tasks, either deal with missing modalities only during evaluation or train separate models to handle specific missing modality settings. In addition, these models are designed for specific tasks, so for example, classification models are not easily adapted to segmentation tasks and vice versa. In this paper, we propose the Shared-Specific Feature Modelling (ShaSpec) method that is considerably simpler and more effective than competing approaches that address the issues above. ShaSpec is designed to take advantage of all available input modalities during training and evaluation by learning shared and specific features to better represent the input data. This is achieved from a strategy that relies on auxiliary tasks based on distribution alignment and domain classification, in addition to a residual feature fusion procedure. Also, the design simplicity of ShaSpec enables its easy adaptation to multiple tasks, such as classification and segmentation. Experiments are conducted on both medical image segmentation and computer vision classification, with results indicating that ShaSpec outperforms competing methods by a large margin. For instance, on BraTS2018, ShaSpec improves the SOTA by more than 3% for enhancing tumour, 5% for tumour core and 3% for whole tumour.

翻译：缺失模态问题是多模态模型面临的关键但难以解决的挑战。当前处理多模态任务中缺失模态问题的方法，要么仅在评估阶段应对缺失模态，要么训练独立模型以处理特定的缺失模态设置。此外，这些模型专为特定任务设计，因此分类模型难以直接迁移至分割任务，反之亦然。本文提出共享-特定特征建模方法（Shared-Specific Feature Modelling, ShaSpec），该方法在解决上述问题时比现有竞争方法更为简单且有效。ShaSpec旨在通过学习共享特征与特定特征以更好地表征输入数据，从而在训练和评估阶段充分利用所有可用输入模态。这一目标通过基于分布对齐与领域分类的辅助任务策略，以及残差特征融合过程实现。此外，ShaSpec设计的简洁性使其易于适配多种任务，如分类与分割。实验涵盖医学图像分割与计算机视觉分类任务，结果表明ShaSpec在性能上大幅超越竞争方法。例如，在BraTS2018数据集上，ShaSpec在增强肿瘤区域、肿瘤核心区域及全肿瘤区域分别将最先进方法（SOTA）提升超过3%、5%与3%。