It is an open secret that ImageNet is treated as the panacea of pretraining. Particularly in medical machine learning, models not trained from scratch are often finetuned based on ImageNet-pretrained models. We posit that pretraining on data from the domain of the downstream task should almost always be preferred instead. We leverage RadNet-12M, a dataset containing more than 12 million computed tomography (CT) image slices, to explore the efficacy of self-supervised pretraining on medical and natural images. Our experiments cover intra- and cross-domain transfer scenarios, varying data scales, finetuning vs. linear evaluation, and feature space analysis. We observe that intra-domain transfer compares favorably to cross-domain transfer, achieving comparable or improved performance (0.44% - 2.07% performance increase using RadNet pretraining, depending on the experiment) and demonstrate the existence of a domain boundary-related generalization gap and domain-specific learned features.

翻译：众所周知，ImageNet被当作预训练的万能灵药。特别是在医学机器学习领域，非从头训练的模型通常基于ImageNet预训练模型进行微调。我们认为，优先使用下游任务领域的数据进行预训练应始终是更好的选择。我们利用RadNet-12M（包含超过1200万张计算机断层扫描（CT）图像切片的数据集）来探索在医学图像和自然图像上进行自监督预训练的有效性。实验涵盖领域内与跨领域迁移场景、不同数据规模、微调与线性评估以及特征空间分析。我们观察到，领域内迁移比跨领域迁移更具优势，可实现相当或更优的性能（根据实验不同，使用RadNet预训练可提升0.44%至2.07%的性能），并证明了存在与领域边界相关的泛化差距以及领域特定的学习特征。