Humans can learn incrementally, whereas neural networks forget previously acquired information catastrophically. Continual Learning (CL) approaches seek to bridge this gap by facilitating the transfer of knowledge to both previous tasks (backward transfer) and future ones (forward transfer) during training. Recent research has shown that self-supervision can produce versatile models that can generalize well to diverse downstream tasks. However, contrastive self-supervised learning (CSSL), a popular self-supervision technique, has limited effectiveness in online CL (OCL). OCL only permits one iteration of the input dataset, and CSSL's low sample efficiency hinders its use on the input data-stream. In this work, we propose Continual Learning via Equivariant Regularization (CLER), an OCL approach that leverages equivariant tasks for self-supervision, avoiding CSSL's limitations. Our method represents the first attempt at combining equivariant knowledge with CL and can be easily integrated with existing OCL methods. Extensive ablations shed light on how equivariant pretext tasks affect the network's information flow and its impact on CL dynamics.

翻译：人类能够逐步学习新知识，而神经网络却会灾难性地遗忘先前获取的信息。持续学习（CL）方法通过促进知识向先前任务（反向迁移）和未来任务（前向迁移）的传递来弥补这一差距。最新研究表明，自监督学习可以产生能够良好泛化至多种下游任务的多功能模型。然而，作为广泛使用的自监督技术，对比自监督学习（CSSL）在在线持续学习（OCL）中效果有限。OCL仅允许对输入数据集进行单次遍历，而CSSL的低样本效率制约了其在输入数据流中的应用。本文提出基于等变正则化的持续学习（CLER）方法，这是一种利用等变任务进行自监督学习的OCL方案，有效规避了CSSL的局限性。本方法首次尝试将等变知识与CL相结合，并可便捷地与现有OCL方法集成。大量消融实验揭示了等变前置任务如何影响网络信息流及其对CL动态过程的作用机制。