While a key component to the success of deep learning is the availability of massive amounts of training data, medical image datasets are often limited in diversity and size. Transfer learning has the potential to bridge the gap between related yet different domains. For medical applications, however, it remains unclear whether it is more beneficial to pre-train on natural or medical images. We aim to shed light on this problem by comparing initialization on ImageNet and RadImageNet on seven medical classification tasks. Our work includes a replication study, which yields results contrary to previously published findings. In our experiments, ResNet50 models pre-trained on ImageNet tend to outperform those trained on RadImageNet. To gain further insights, we investigate the learned representations using Canonical Correlation Analysis (CCA) and compare the predictions of the different models. Our results indicate that, contrary to intuition, ImageNet and RadImageNet may converge to distinct intermediate representations, which appear to diverge further during fine-tuning. Despite these distinct representations, the predictions of the models remain similar. Our findings show that the similarity between networks before and after fine-tuning does not correlate with performance gains, suggesting that the advantages of transfer learning might not solely originate from the reuse of features in the early layers of a convolutional neural network.

翻译：尽管深度学习成功的关键因素之一是海量训练数据的可用性，但医学影像数据集在多样性和规模方面往往有限。迁移学习有望弥合相关但不同领域之间的差距。然而，对于医学应用而言，尚不明确在自然图像还是医学图像上进行预训练更为有益。我们旨在通过比较ImageNet和RadImageNet在七个医学分类任务上的初始化效果来阐明这一问题。我们的工作包含一项重复性研究，其得出了与先前已发表研究相反的结论。在我们的实验中，基于ImageNet预训练的ResNet50模型的表现通常优于基于RadImageNet预训练的模型。为了获得更深入的见解，我们使用典型相关分析（CCA）研究学习到的表征，并比较不同模型的预测结果。我们的结果表明，与直觉相反，ImageNet和RadImageNet可能收敛到不同的中间表征，并且这些表征在微调过程中似乎进一步分化。尽管表征存在差异，但模型的预测结果仍然相似。我们的研究发现表明，网络在微调前和微调后的相似性与性能提升无相关性，这意味着迁移学习的优势可能并非仅仅源于卷积神经网络早期层中特征的复用。