Continuous unsupervised representation learning (CURL) research has greatly benefited from improvements in self-supervised learning (SSL) techniques. As a result, existing CURL methods using SSL can learn high-quality representations without any labels, but with a notable performance drop when learning on a many-tasks data stream. We hypothesize that this is caused by the regularization losses that are imposed to prevent forgetting, leading to a suboptimal plasticity-stability trade-off: they either do not adapt fully to the incoming data (low plasticity), or incur significant forgetting when allowed to fully adapt to a new SSL pretext-task (low stability). In this work, we propose to train an expert network that is relieved of the duty of keeping the previous knowledge and can focus on performing optimally on the new tasks (optimizing plasticity). In the second phase, we combine this new knowledge with the previous network in an adaptation-retrospection phase to avoid forgetting and initialize a new expert with the knowledge of the old network. We perform several experiments showing that our proposed approach outperforms other CURL exemplar-free methods in few- and many-task split settings. Furthermore, we show how to adapt our approach to semi-supervised continual learning (Semi-SCL) and show that we surpass the accuracy of other exemplar-free Semi-SCL methods and reach the results of some others that use exemplars.

翻译：连续无监督表示学习（CURL）研究极大受益于自监督学习（SSL）技术的进步。因此，现有基于SSL的CURL方法能够在不依赖标签的情况下学习高质量表示，但在多任务数据流中进行学习时会出现显著性能下降。我们假设这是由于为防止遗忘而施加的正则化损失导致了次优的塑性-稳定性权衡：模型要么无法完全适应新数据（低塑性），要么在允许完全适应新SSL预文本任务时产生严重遗忘（低稳定性）。本文提出训练一个无需承担保留先前知识职责的专家网络，使其能够专注于优化新任务性能（优化塑性）。在第二阶段中，我们通过适应-回顾机制将该新知识与原有网络结合以避免遗忘，并以旧网络的知识初始化新专家。通过多项实验表明，所提方法在少任务与多任务分割设置中均优于其他无样本CURL方法。此外，我们展示了如何将该方法适配至半监督持续学习（Semi-SCL）场景，其准确率超越其他无样本Semi-SCL方法，甚至达到部分使用样本的方法的性能水平。