The principle underlying most existing continual learning (CL) methods is to prioritize stability by penalizing changes in parameters crucial to old tasks, while allowing for plasticity in other parameters. The importance of weights for each task can be determined either explicitly through learning a task-specific mask during training (e.g., parameter isolation-based approaches) or implicitly by introducing a regularization term (e.g., regularization-based approaches). However, all these methods assume that the importance of weights for each task is unknown prior to data exposure. In this paper, we propose ScrollNet as a scrolling neural network for continual learning. ScrollNet can be seen as a dynamic network that assigns the ranking of weight importance for each task before data exposure, thus achieving a more favorable stability-plasticity tradeoff during sequential task learning by reassigning this ranking for different tasks. Additionally, we demonstrate that ScrollNet can be combined with various CL methods, including regularization-based and replay-based approaches. Experimental results on CIFAR100 and TinyImagenet datasets show the effectiveness of our proposed method. We release our code at https://github.com/FireFYF/ScrollNet.git.

翻译：大多数现有持续学习方法的基本原理是通过惩罚对旧任务至关重要的参数变化来优先保证稳定性，同时允许其他参数具有可塑性。每个任务权重的重要性既可以通过在训练期间学习特定任务的掩码来显式确定（例如基于参数隔离的方法），也可以通过引入正则化项来隐式确定（例如基于正则化的方法）。然而，所有这些方法都假设在接触数据之前，每个任务权重的重要性是未知的。本文提出了ScrollNet，一种用于持续学习的滚动神经网络。ScrollNet可被视为动态网络，它能在接触数据之前为每个任务分配权重重要性的排序，从而通过为不同任务重新分配该排序，在序列任务学习过程中实现更优的稳定性-可塑性权衡。此外，我们证明ScrollNet可与多种持续学习方法结合，包括基于正则化的方法和基于重放的方法。在CIFAR100和TinyImagenet数据集上的实验结果表明了我们提出的方法的有效性。我们已在https://github.com/FireFYF/ScrollNet.git发布代码。