A common approach to transfer learning under distribution shift is to fine-tune the last few layers of a pre-trained model, preserving learned features while also adapting to the new task. This paper shows that in such settings, selectively fine-tuning a subset of layers (which we term surgical fine-tuning) matches or outperforms commonly used fine-tuning approaches. Moreover, the type of distribution shift influences which subset is more effective to tune: for example, for image corruptions, fine-tuning only the first few layers works best. We validate our findings systematically across seven real-world data tasks spanning three types of distribution shifts. Theoretically, we prove that for two-layer neural networks in an idealized setting, first-layer tuning can outperform fine-tuning all layers. Intuitively, fine-tuning more parameters on a small target dataset can cause information learned during pre-training to be forgotten, and the relevant information depends on the type of shift.

翻译：分布偏移下迁移学习的常见方法是微调预训练模型的最后几层，在适应新任务的同时保留已学特征。本文表明，在此类场景中，选择性微调部分层（称为手术微调）的表现可与常用微调方法相当或更优。此外，分布偏移的类型会影响最有效的微调子集：例如，针对图像损坏，仅微调前几层效果最佳。我们通过涵盖三种分布偏移类型的七个真实数据任务系统验证了上述发现。理论上，我们证明在理想化设置下，针对两层神经网络，首层微调可优于全层微调。直观而言，在小型目标数据集上微调更多参数会导致预训练所学信息被遗忘，且相关信息取决于偏移类型。