Transfer learning has emerged as a key approach in the machine learning domain, enabling the application of knowledge derived from one domain to improve performance on subsequent tasks. Given the often limited information about these subsequent tasks, a strong transfer learning approach calls for the model to capture a diverse range of features during the initial pretraining stage. However, recent research suggests that, without sufficient regularization, the network tends to concentrate on features that primarily reduce the pretraining loss function. This tendency can result in inadequate feature learning and impaired generalization capability for target tasks. To address this issue, we propose Variance-Covariance Regularization (VCR), a regularization technique aimed at fostering diversity in the learned network features. Drawing inspiration from recent advancements in the self-supervised learning approach, our approach promotes learned representations that exhibit high variance and minimal covariance, thus preventing the network from focusing solely on loss-reducing features. We empirically validate the efficacy of our method through comprehensive experiments coupled with in-depth analytical studies on the learned representations. In addition, we develop an efficient implementation strategy that assures minimal computational overhead associated with our method. Our results indicate that VCR is a powerful and efficient method for enhancing transfer learning performance for both supervised learning and self-supervised learning, opening new possibilities for future research in this domain.

翻译：迁移学习已成为机器学习领域的一项关键方法，使得一个领域学到的知识能够被应用于提升后续任务的表现。鉴于后续任务的信息通常有限，强大的迁移学习方法要求模型在初始预训练阶段捕获多样化的特征。然而，近期研究表明，若无充分正则化，网络倾向于关注主要降低预训练损失函数的特征。这种倾向可能导致特征学习不足，并损害目标任务的泛化能力。为解决此问题，我们提出方差-协方差正则化（VCR），一种旨在促进所学网络特征多样性的正则化技术。受自监督学习方法最新进展的启发，我们的方法促进学习表示具有高方差和低协方差，从而防止网络仅聚焦于降低损失的特征。我们通过综合实验及对学习表示的深度分析研究，实证验证了该方法的有效性。此外，我们开发了一种高效实现策略，确保该方法带来最小的计算开销。结果表明，VCR是一种强大且高效的方法，能够提升监督学习和自监督学习的迁移学习性能，为该领域的未来研究开辟了新可能。