Models initialized from self-supervised pretraining may suffer from poor alignment with downstream tasks, reducing the extent to which subsequent fine-tuning can adapt pretrained features toward downstream objectives. To mitigate this, we introduce BiSSL, a novel bilevel training framework that enhances the alignment of self-supervised pretrained models with downstream tasks prior to fine-tuning. BiSSL acts as an intermediate training stage conducted after conventional self-supervised pretraining and is tasked with solving a bilevel optimization problem that incorporates the pretext and downstream training objectives in its lower- and upper-level objectives, respectively. This approach explicitly models the interdependence between the pretraining and fine-tuning stages within the conventional self-supervised learning pipeline, facilitating enhanced information sharing between them that ultimately leads to a model initialization better aligned with the downstream task. We propose a general training algorithm for BiSSL that is compatible with a broad range of pretext and downstream tasks. Using SimCLR and Bootstrap Your Own Latent to pretrain ResNet-50 backbones on the ImageNet dataset, we demonstrate that our proposed framework significantly improves accuracy on the vast majority of 12 downstream image classification datasets, as well as on object detection. Exploratory analyses alongside investigative experiments further provide compelling evidence that BiSSL enhances downstream alignment.

翻译：从自监督预训练初始化的模型可能与下游任务存在较差的对齐性，从而降低了后续微调将预训练特征适配到下游目标的程度。为缓解此问题，我们提出了BiSSL，一种新颖的双层训练框架，可在微调前增强自监督预训练模型与下游任务的对齐。BiSSL作为常规自监督预训练后进行的中间训练阶段，其任务是求解一个双层优化问题，该问题分别在其下层和上层目标中包含了代理任务与下游训练目标。该方法在常规自监督学习流程中显式地建模了预训练与微调阶段之间的相互依赖关系，促进了二者间增强的信息共享，最终得到与下游任务更对齐的模型初始化。我们提出了一种适用于BiSSL的通用训练算法，该算法兼容广泛的代理任务与下游任务。通过在ImageNet数据集上使用SimCLR和Bootstrap Your Own Latent对ResNet-50骨干网络进行预训练，我们证明了所提框架在12个下游图像分类数据集中的绝大多数上显著提升了准确率，并在目标检测任务上同样有效。探索性分析与实验研究进一步提供了有力证据，表明BiSSL确实增强了下游对齐性。