Inspired by the success of Self-supervised learning (SSL) in learning visual representations from unlabeled data, a few recent works have studied SSL in the context of continual learning (CL), where multiple tasks are learned sequentially, giving rise to a new paradigm, namely self-supervised continual learning (SSCL). It has been shown that the SSCL outperforms supervised continual learning (SCL) as the learned representations are more informative and robust to catastrophic forgetting. However, if not designed intelligently, the training complexity of SSCL may be prohibitively high due to the inherent training cost of SSL. In this work, by investigating the task correlations in SSCL setup first, we discover an interesting phenomenon that, with the SSL-learned background model, the intermediate features are highly correlated between tasks. Based on this new finding, we propose a new SSCL method with layer-wise freezing which progressively freezes partial layers with the highest correlation ratios for each task to improve training computation efficiency and memory efficiency. Extensive experiments across multiple datasets are performed, where our proposed method shows superior performance against the SoTA SSCL methods under various SSL frameworks. For example, compared to LUMP, our method achieves 12\%/14\%/12\% GPU training time reduction, 23\%/26\%/24\% memory reduction, 35\%/34\%/33\% backward FLOPs reduction, and 1.31\%/1.98\%/1.21\% forgetting reduction without accuracy degradation on three datasets, respectively.

翻译：受自监督学习（SSL）从无标签数据中学习视觉表征成功的启发，近期部分研究将SSL应用于持续学习（CL）场景（任务需按序学习），催生了自监督持续学习（SSCL）这一新范式。研究表明，SSCL由于学习到的表征更具信息量且对灾难性遗忘更具鲁棒性，其性能优于监督持续学习（SCL）。然而，若缺乏智能设计，SSCL的训练复杂度可能因SSL固有的训练成本而过高。本文首先探究SSCL框架下的任务相关性，发现一个有趣现象：采用SSL预训练的基底模型时，不同任务间的中间特征高度相关。基于这一新发现，我们提出一种逐层冻结的SSCL方法——对每个任务渐进冻结相关性比例最高的部分网络层，从而提升训练计算效率与内存效率。在多个数据集上的大量实验表明，所提方法在各种SSL框架下均展现出优于现有最佳SSCL方法的性能。例如，与LUMP方法相比，本方法在三个数据集上分别实现了：GPU训练时间减少12%/14%/12%，内存占用减少23%/26%/24%，反向传播计算量（FLOPs）降低35%/34%/33%，遗忘率降低1.31%/1.98%/1.21%，且未造成精度损失。