While a key component to the success of deep learning is the availability of massive amounts of training data, medical image datasets are often limited in diversity and size. Transfer learning has the potential to bridge the gap between related yet different domains. For medical applications, however, it remains unclear whether it is more beneficial to pre-train on natural or medical images. We aim to shed light on this problem by comparing initialization on ImageNet and RadImageNet on seven medical classification tasks. Our work includes a replication study, which yields results contrary to previously published findings. In our experiments, ResNet50 models pre-trained on ImageNet tend to outperform those trained on RadImageNet. To gain further insights, we investigate the learned representations using Canonical Correlation Analysis (CCA) and compare the predictions of the different models. Our results indicate that, contrary to intuition, ImageNet and RadImageNet may converge to distinct intermediate representations, which appear to diverge further during fine-tuning. Despite these distinct representations, the predictions of the models remain similar. Our findings show that the similarity between networks before and after fine-tuning does not correlate with performance gains, suggesting that the advantages of transfer learning might not solely originate from the reuse of features in the early layers of a convolutional neural network.

翻译：尽管深度学习成功的关键因素之一在于海量训练数据的可获得性，但医学影像数据集通常在多样性和规模上存在限制。迁移学习具有弥补相关但不同领域之间差距的潜力。然而，对于医学应用而言，尚不清楚是使用自然图像还是医学图像进行预训练更为有利。我们旨在通过比较在ImageNet和RadImageNet上初始化的七项医学分类任务来阐明这一问题。本研究包含一项重复性实验，其结果与已发表的结论相悖。实验中，基于ImageNet预训练的ResNet50模型通常优于基于RadImageNet训练的模型。为获得更深入的见解，我们利用典型相关分析（CCA）探究学到的表征，并比较不同模型的预测结果。结果表明，与直觉相反，ImageNet和RadImageNet可能收敛至不同的中间表征，而这些表征在微调过程中似乎进一步分化。尽管表征存在差异，模型的预测结果仍保持相似。我们的发现表明，微调前后网络间的相似度与性能提升无关，这意味着迁移学习的优势可能并非单纯源于卷积神经网络浅层特征的复用。