Self-supervised learning is an efficient pre-training method for medical image analysis. However, current research is mostly confined to specific-modality data pre-training, consuming considerable time and resources without achieving universality across different modalities. A straightforward solution is combining all modality data for joint self-supervised pre-training, which poses practical challenges. Firstly, our experiments reveal conflicts in representation learning as the number of modalities increases. Secondly, multi-modal data collected in advance cannot cover all real-world scenarios. In this paper, we reconsider versatile self-supervised learning from the perspective of continual learning and propose MedCoSS, a continuous self-supervised learning approach for multi-modal medical data. Unlike joint self-supervised learning, MedCoSS assigns different modality data to different training stages, forming a multi-stage pre-training process. To balance modal conflicts and prevent catastrophic forgetting, we propose a rehearsal-based continual learning method. We introduce the k-means sampling strategy to retain data from previous modalities and rehearse it when learning new modalities. Instead of executing the pretext task on buffer data, a feature distillation strategy and an intra-modal mixup strategy are applied to these data for knowledge retention. We conduct continuous self-supervised pre-training on a large-scale multi-modal unlabeled dataset, including clinical reports, X-rays, CT scans, MRI scans, and pathological images. Experimental results demonstrate MedCoSS's exceptional generalization ability across nine downstream datasets and its significant scalability in integrating new modality data. Code and pre-trained weight are available at https://github.com/yeerwen/MedCoSS.

翻译：自监督学习是医学图像分析中高效的预训练方法。然而，当前研究多局限于特定模态数据的预训练，不仅消耗大量时间和资源，且无法实现跨模态的通用性。一个直接的解决方案是联合所有模态数据进行自监督预训练，但这面临实际挑战。首先，我们的实验表明，随着模态数量增加，表示学习会产生冲突。其次，预先收集的多模态数据无法覆盖所有现实场景。本文从持续学习的视角重新审视多用途自监督学习，提出MedCoSS——一种面向多模态医学数据的连续自监督学习方法。与联合自监督学习不同，MedCoSS将不同模态数据分配至不同训练阶段，形成多阶段预训练过程。为平衡模态冲突并防止灾难性遗忘，我们提出基于重演的持续学习方法，引入k-means采样策略保留先前模态数据，并在学习新模态时进行重演。相比对缓冲数据直接执行前置任务，我们采用特征蒸馏策略和模态内混合策略来保留知识。我们在大规模多模态无标注数据集（包括临床报告、X光片、CT扫描、MRI扫描和病理图像）上开展连续自监督预训练。实验结果表明，MedCoSS在九个下游数据集上展现出卓越的泛化能力，并在整合新模态数据时具有显著可扩展性。代码和预训练权重已开源至https://github.com/yeerwen/MedCoSS。