Self-supervised learning makes significant progress in pre-training large models, but struggles with small models. Previous solutions to this problem rely mainly on knowledge distillation, which involves a two-stage procedure: first training a large teacher model and then distilling it to improve the generalization ability of smaller ones. In this work, we present a one-stage solution to obtain pre-trained small models without the need for extra teachers, namely, slimmable networks for contrastive self-supervised learning (\emph{SlimCLR}). A slimmable network consists of a full network and several weight-sharing sub-networks, which can be pre-trained once to obtain various networks, including small ones with low computation costs. However, interference between weight-sharing networks leads to severe performance degradation in self-supervised cases, as evidenced by \emph{gradient magnitude imbalance} and \emph{gradient direction divergence}. The former indicates that a small proportion of parameters produce dominant gradients during backpropagation, while the main parameters may not be fully optimized. The latter shows that the gradient direction is disordered, and the optimization process is unstable. To address these issues, we introduce three techniques to make the main parameters produce dominant gradients and sub-networks have consistent outputs. These techniques include slow start training of sub-networks, online distillation, and loss re-weighting according to model sizes. Furthermore, theoretical results are presented to demonstrate that a single slimmable linear layer is sub-optimal during linear evaluation. Thus a switchable linear probe layer is applied during linear evaluation. We instantiate SlimCLR with typical contrastive learning frameworks and achieve better performance than previous arts with fewer parameters and FLOPs.

翻译：自监督学习在预训练大模型方面取得了显著进展，但在小模型上仍面临挑战。以往解决该问题的方法主要依赖知识蒸馏，即先训练一个大教师模型，再通过蒸馏提升小模型的泛化能力，这需要两阶段流程。本文提出一种无需额外教师模型、可单阶段获取预训练小模型的方案，即用于对比自监督学习的可精简网络（SlimCLR）。可精简网络由一个完整网络和多个共享权重的子网络组成，可一次性预训练获得多种网络，包括计算成本低的小模型。然而，共享权重网络间的干扰会导致自监督场景下性能严重下降，表现为梯度幅度失衡和梯度方向发散。前者指反向传播中少数参数产生主导梯度，而主要参数可能未充分优化；后者表明梯度方向混乱，优化过程不稳定。为解决这些问题，我们引入三项技术，使主要参数生成主导梯度并确保子网络输出一致性，包括子网络慢启动训练、在线蒸馏和按模型尺寸调整损失权重。此外，理论结果表明单一可精简线性层在线性评估中非最优，因此在线性评估阶段采用可切换线性探测层。我们基于典型对比学习框架实例化SlimCLR，在更少的参数量和FLOPs下取得了优于现有方法的性能。