Transfer learning plays a key role in advancing machine learning models, yet conventional supervised pretraining often undermines feature transferability by prioritizing features that minimize the pretraining loss. In this work, we adapt a self-supervised learning regularization technique from the VICReg method to supervised learning contexts, introducing Variance-Covariance Regularization (VCReg). This adaptation encourages the network to learn high-variance, low-covariance representations, promoting learning more diverse features. We outline best practices for an efficient implementation of our framework, including applying it to the intermediate representations. Through extensive empirical evaluation, we demonstrate that our method significantly enhances transfer learning for images and videos, achieving state-of-the-art performance across numerous tasks and datasets. VCReg also improves performance in scenarios like long-tail learning and hierarchical classification. Additionally, we show its effectiveness may stem from its success in addressing challenges like gradient starvation and neural collapse. In summary, VCReg offers a universally applicable regularization framework that significantly advances transfer learning and highlights the connection between gradient starvation, neural collapse, and feature transferability.

翻译：迁移学习在推动机器学习模型发展中发挥关键作用，然而传统的有监督预训练往往通过优先选择最小化预训练损失的特征来削弱特征的可迁移性。在本工作中，我们将VICReg方法中的自监督学习正则化技术适配至有监督学习场景，引入方差-协方差正则化（VCReg）。该适配促使网络学习高方差、低协方差的表示，从而促进更多样化特征的学习。我们概述了高效实现该框架的最佳实践，包括将其应用于中间表示层。通过大量实证评估，我们证明该方法显著提升了图像与视频的迁移学习性能，在众多任务和数据集上达到领先水平。VCReg在长尾学习与层次分类等场景中也展现出性能提升。此外，其有效性可能源于成功应对梯度饥饿和神经坍缩等挑战。总之，VCReg提供了一个普遍适用的正则化框架，显著推进了迁移学习发展，并揭示了梯度饥饿、神经坍缩与特征可迁移性之间的内在关联。