Continual learning aims to emulate the human ability to continually accumulate knowledge over sequential tasks. The main challenge is to maintain performance on previously learned tasks after learning new tasks, i.e., to avoid catastrophic forgetting. We propose a Channel-wise Lightweight Reprogramming (CLR) approach that helps convolutional neural networks (CNNs) overcome catastrophic forgetting during continual learning. We show that a CNN model trained on an old task (or self-supervised proxy task) could be ``reprogrammed" to solve a new task by using our proposed lightweight (very cheap) reprogramming parameter. With the help of CLR, we have a better stability-plasticity trade-off to solve continual learning problems: To maintain stability and retain previous task ability, we use a common task-agnostic immutable part as the shared ``anchor" parameter set. We then add task-specific lightweight reprogramming parameters to reinterpret the outputs of the immutable parts, to enable plasticity and integrate new knowledge. To learn sequential tasks, we only train the lightweight reprogramming parameters to learn each new task. Reprogramming parameters are task-specific and exclusive to each task, which makes our method immune to catastrophic forgetting. To minimize the parameter requirement of reprogramming to learn new tasks, we make reprogramming lightweight by only adjusting essential kernels and learning channel-wise linear mappings from anchor parameters to task-specific domain knowledge. We show that, for general CNNs, the CLR parameter increase is less than 0.6\% for any new task. Our method outperforms 13 state-of-the-art continual learning baselines on a new challenging sequence of 53 image classification datasets. Code and data are available at https://github.com/gyhandy/Channel-wise-Lightweight-Reprogramming

翻译：持续学习旨在模拟人类在连续任务中持续积累知识的能力。其主要挑战在于学习新任务后，仍能维持先前学习任务的表现，即避免灾难性遗忘。我们提出了一种通道级轻量重编程（CLR）方法，帮助卷积神经网络（CNN）在持续学习过程中克服灾难性遗忘问题。研究表明，通过使用我们提出的轻量（极低成本）重编程参数，一个在旧任务（或自监督代理任务）上训练的CNN模型可以被“重编程”以解决新任务。借助CLR，我们能够实现更优的稳定性-可塑性权衡来解决持续学习问题：为保持稳定性并保留先前任务能力，我们将一个通用的任务无关不可变部分作为共享的“锚点”参数集。随后，我们添加任务特定的轻量重编程参数来重新解释不可变部分的输出，以增强可塑性并整合新知识。为了学习连续任务，我们仅训练轻量重编程参数来学习每个新任务。重编程参数是任务特定且独占的，这使得我们的方法能够免疫灾难性遗忘。为最小化学习新任务所需的参数开销，我们仅调整关键卷积核并学习从锚点参数到任务特定领域知识的通道级线性映射，从而实现重编程的轻量化。实验表明，对于通用CNN，任何新任务所需的CLR参数增量均低于0.6%。在由53个图像分类数据集构成的全新挑战性序列上，我们的方法优于13种当前最先进的持续学习基线方法。代码与数据已开源至https://github.com/gyhandy/Channel-wise-Lightweight-Reprogramming。