We consider transfer learning approaches that fine-tune a pretrained deep neural network on a target task. We study the generalization properties of fine-tuning to understand the problem of overfitting, which commonly occurs in practice. Previous works have shown that constraining the distance from the initialization of fine-tuning improves generalization. Using a PAC-Bayesian analysis, we observe that besides distance from initialization, Hessians affect generalization through the noise stability of deep neural networks against noise injections. Motivated by the observation, we develop Hessian distance-based generalization bounds for a wide range of fine-tuning methods. Additionally, we study the robustness of fine-tuning in the presence of noisy labels. We design an algorithm incorporating consistent losses and distance-based regularization for fine-tuning, along with a generalization error guarantee under class conditional independent noise in the training set labels. We perform a detailed empirical study of our algorithm on various noisy environments and architectures. On six image classification tasks whose training labels are generated with programmatic labeling, we find a 3.26% accuracy gain over prior fine-tuning methods. Meanwhile, the Hessian distance measure of the fine-tuned model decreases by six times more than existing approaches.

翻译：我们考虑在目标任务上对预训练深度神经网络进行微调的迁移学习方法。为理解实践中常见的过拟合问题，本文研究了微调的泛化特性。先前研究表明，限制微调过程与初始化参数的距离有助于提升泛化性能。通过PAC-Bayesian分析，我们发现除初始化距离外，Hessian矩阵通过深度神经网络对噪声注入的噪声稳定性影响泛化性能。基于这一发现，我们为多种微调方法建立了基于Hessian距离的泛化界。此外，我们研究了含噪标签场景下微调的鲁棒性，设计了一种融合一致性损失与距离正则化的微调算法，并给出了训练集标签在类别条件独立噪声下的泛化误差保证。我们在多种噪声环境及网络架构上对该算法进行了详尽的实证研究。在六个训练标签通过程序化标注生成的图像分类任务中，相较于现有微调方法，我们的算法取得了3.26%的准确率提升。同时，微调后模型的Hessian距离度量指标较现有方法降低了六倍。