Obtaining large pre-trained models that can be fine-tuned to new tasks with limited annotated samples has remained an open challenge for medical imaging data. While pre-trained deep networks on ImageNet and vision-language foundation models trained on web-scale data are prevailing approaches, their effectiveness on medical tasks is limited due to the significant domain shift between natural and medical images. To bridge this gap, we introduce LVM-Med, the first family of deep networks trained on large-scale medical datasets. We have collected approximately 1.3 million medical images from 55 publicly available datasets, covering a large number of organs and modalities such as CT, MRI, X-ray, and Ultrasound. We benchmark several state-of-the-art self-supervised algorithms on this dataset and propose a novel self-supervised contrastive learning algorithm using a graph-matching formulation. The proposed approach makes three contributions: (i) it integrates prior pair-wise image similarity metrics based on local and global information; (ii) it captures the structural constraints of feature embeddings through a loss function constructed via a combinatorial graph-matching objective; and (iii) it can be trained efficiently end-to-end using modern gradient-estimation techniques for black-box solvers. We thoroughly evaluate the proposed LVM-Med on 15 downstream medical tasks ranging from segmentation and classification to object detection, and both for the in and out-of-distribution settings. LVM-Med empirically outperforms a number of state-of-the-art supervised, self-supervised, and foundation models. For challenging tasks such as Brain Tumor Classification or Diabetic Retinopathy Grading, LVM-Med improves previous vision-language models trained on 1 billion masks by 6-7% while using only a ResNet-50.

翻译：获取能够通过有限标注样本微调至新任务的大型预训练模型，仍是医学影像数据领域面临的开放挑战。尽管基于ImageNet预训练的深度网络和基于网络规模数据训练的视觉语言基础模型是主流方法，但由于自然图像与医学图像之间存在显著域偏移，这些方法在医学任务中的有效性受到限制。为弥合这一差距，我们提出LVM-Med——首个在大规模医学数据集上训练的深度网络系列。我们从55个公开数据集中收集约130万张医学图像，涵盖CT、MRI、X射线和超声等多种器官与模态。我们在该数据集上基准测试了多项先进的自监督算法，并提出一种基于图匹配公式的新型自监督对比学习算法。该方法有三项贡献：(i) 整合基于局部与全局信息的先验成对图像相似度度量；(ii) 通过基于组合图匹配目标的损失函数捕获特征嵌入的结构约束；(iii) 利用黑盒求解器的现代梯度估计技术实现高效端到端训练。我们在15项下游医学任务（涵盖分割、分类及目标检测）上全面评估LVM-Med，包括分布内与分布外场景。实验表明，LVM-Med在性能上优于多项先进的有监督、自监督及基础模型。在如脑肿瘤分类或糖尿病视网膜病变分级等具有挑战性的任务中，LVM-Med在仅使用ResNet-50的情况下，相较于基于10亿掩码训练的视觉语言模型，性能提升6-7%。