While a key component to the success of deep learning is the availability of massive amounts of training data, medical image datasets are often limited in diversity and size. Transfer learning has the potential to bridge the gap between related yet different domains. For medical applications, however, it remains unclear whether it is more beneficial to pre-train on natural or medical images. We aim to shed light on this problem by comparing initialization on ImageNet and RadImageNet on seven medical classification tasks. We investigate their learned representations with Canonical Correlation Analysis (CCA) and compare the predictions of the different models. We find that overall the models pre-trained on ImageNet outperform those trained on RadImageNet. Our results show that, contrary to intuition, ImageNet and RadImageNet converge to distinct intermediate representations, and that these representations are even more dissimilar after fine-tuning. Despite these distinct representations, the predictions of the models remain similar. Our findings challenge the notion that transfer learning is effective due to the reuse of general features in the early layers of a convolutional neural network and show that weight similarity before and after fine-tuning is negatively related to performance gains.

翻译：尽管深度学习成功的关键因素之一是大量训练数据的可用性，医学影像数据集在多样性和规模上常常受到限制。迁移学习有潜力弥合相关但不同领域之间的差距。然而，对于医学应用而言，尚不清楚在自然图像还是医学图像上进行预训练更为有利。我们旨在通过比较在ImageNet和RadImageNet上初始化的七个医学分类任务来阐明这一问题。我们利用典型相关分析（CCA）研究它们习得的表征，并比较不同模型的预测结果。我们发现，总体而言，在ImageNet上预训练的模型优于在RadImageNet上训练的模型。我们的结果表明，与直觉相反，ImageNet和RadImageNet收敛到不同的中间表征，且这些表征在微调后甚至更加不相似。尽管这些表征存在差异，但模型的预测结果仍然相似。我们的发现挑战了迁移学习因重用卷积神经网络早期层的通用特征而有效的观点，并显示微调前后的权重相似性与性能提升呈负相关。