Transfer learning is a powerful tool enabling model training with limited amounts of data. This technique is particularly useful in real-world problems where data availability is often a serious limitation. The simplest transfer learning protocol is based on ``freezing" the feature-extractor layers of a network pre-trained on a data-rich source task, and then adapting only the last layers to a data-poor target task. This workflow is based on the assumption that the feature maps of the pre-trained model are qualitatively similar to the ones that would have been learned with enough data on the target task. In this work, we show that this protocol is often sub-optimal, and the largest performance gain may be achieved when smaller portions of the pre-trained network are kept frozen. In particular, we make use of a controlled framework to identify the optimal transfer depth, which turns out to depend non-trivially on the amount of available training data and on the degree of source-target task correlation. We then characterize transfer optimality by analyzing the internal representations of two networks trained from scratch on the source and the target task through multiple established similarity measures.

翻译：迁移学习是一种强大的工具，能够利用有限数据训练模型。在数据可用性常成为严重限制的现实问题中，该技术尤为有用。最简单的迁移学习协议基于“冻结”在数据丰富的源任务上预训练网络的特征提取层，仅调整最后几层以适应数据匮乏的目标任务。该流程假设预训练模型的特征映射与在目标任务上利用充足数据学习得到的特征映射在性质上相似。本文表明，该协议通常并非最优，当预训练网络的较小部分保持冻结时，可能实现最大的性能提升。具体而言，我们利用受控框架确定最优迁移深度，该深度非平凡地依赖于可用训练数据量及源-目标任务相关性程度。随后，我们通过多种既定相似性度量，分析从零分别在源任务和目标任务上训练的两个网络的内部表征，从而刻画迁移最优性。