Unpaired Multi-Modal Learning (UMML) which leverages unpaired multi-modal data to boost model performance on each individual modality has attracted a lot of research interests in medical image analysis. However, existing UMML methods require multi-modal datasets to be fully labeled, which incurs tremendous annotation cost. In this paper, we investigate the use of partially labeled data for label-efficient unpaired multi-modal learning, which can reduce the annotation cost by up to one half. We term the new learning paradigm as Partially Supervised Unpaired Multi-Modal Learning (PSUMML) and propose a novel Decomposed partial class adaptation with snapshot Ensembled Self-Training (DEST) framework for it. Specifically, our framework consists of a compact segmentation network with modality specific normalization layers for learning with partially labeled unpaired multi-modal data. The key challenge in PSUMML lies in the complex partial class distribution discrepancy due to partial class annotation, which hinders effective knowledge transfer across modalities. We theoretically analyze this phenomenon with a decomposition theorem and propose a decomposed partial class adaptation technique to precisely align the partially labeled classes across modalities to reduce the distribution discrepancy. We further propose a snapshot ensembled self-training technique to leverage the valuable snapshot models during training to assign pseudo-labels to partially labeled pixels for self-training to boost model performance. We perform extensive experiments under different scenarios of PSUMML for two medical image segmentation tasks, namely cardiac substructure segmentation and abdominal multi-organ segmentation. Our framework outperforms existing methods significantly.

翻译：非配对多模态学习通过利用非配对的多模态数据来提升各单一模态上的模型性能，已在医学图像分析领域引起了广泛的研究兴趣。然而，现有的非配对多模态学习方法要求多模态数据集完全标注，这带来了巨大的标注成本。本文研究了使用部分标注数据进行标签高效的非配对多模态学习，该方法可将标注成本降低多达一半。我们将这一新的学习范式称为部分监督非配对多模态学习，并为其提出了一种新颖的分解式部分类别适应与快照集成自训练框架。具体而言，我们的框架包含一个配备模态特定归一化层的紧凑分割网络，用于处理部分标注的非配对多模态数据。部分监督非配对多模态学习中的关键挑战在于由部分类别标注导致的复杂部分类别分布差异，这阻碍了跨模态的有效知识迁移。我们通过一个分解定理从理论上分析了这一现象，并提出了一种分解式部分类别适应技术，以精确对齐跨模态的部分标注类别，从而减少分布差异。我们进一步提出了一种快照集成自训练技术，利用训练过程中有价值的快照模型为部分标注的像素分配伪标签进行自训练，以提升模型性能。我们在部分监督非配对多模态学习的不同场景下，针对心脏子结构分割和腹部多器官分割这两个医学图像分割任务进行了大量实验。我们的框架显著优于现有方法。