Building learning agents that can progressively learn and accumulate knowledge is the core goal of the continual learning (CL) research field. Unfortunately, training a model on new data usually compromises the performance on past data. In the CL literature, this effect is referred to as catastrophic forgetting (CF). CF has been largely studied, and a plethora of methods have been proposed to address it on short sequences of non-overlapping tasks. In such setups, CF always leads to a quick and significant drop in performance in past tasks. Nevertheless, despite CF, recent work showed that SGD training on linear models accumulates knowledge in a CL regression setup. This phenomenon becomes especially visible when tasks reoccur. We might then wonder if DNNs trained with SGD or any standard gradient-based optimization accumulate knowledge in such a way. Such phenomena would have interesting consequences for applying DNNs to real continual scenarios. Indeed, standard gradient-based optimization methods are significantly less computationally expensive than existing CL algorithms. In this paper, we study the progressive knowledge accumulation (KA) in DNNs trained with gradient-based algorithms in long sequences of tasks with data re-occurrence. We propose a new framework, SCoLe (Scaling Continual Learning), to investigate KA and discover that catastrophic forgetting has a limited effect on DNNs trained with SGD. When trained on long sequences with data sparsely re-occurring, the overall accuracy improves, which might be counter-intuitive given the CF phenomenon. We empirically investigate KA in DNNs under various data occurrence frequencies and propose simple and scalable strategies to increase knowledge accumulation in DNNs.

翻译：构建能够逐步学习并积累知识的学习智能体是持续学习（CL）研究领域的核心目标。不幸的是，在新数据上训练模型通常会损害其在旧数据上的性能。在持续学习文献中，这种效应被称为灾难性遗忘（CF）。灾难性遗忘已被广泛研究，并提出了大量方法来解决短序列非重叠任务中的这一问题。在此类设定中，灾难性遗忘总是导致过去任务的性能快速且显著下降。然而，尽管存在灾难性遗忘，近期研究表明，在持续学习回归设定下，线性模型的随机梯度下降（SGD）训练能够积累知识。当任务重复出现时，这一现象尤为明显。我们或许会思考：使用SGD或任何标准基于梯度的优化方法训练的深度神经网络（DNN）是否也能以类似方式积累知识？此类现象将对深度神经网络在真实持续场景中的应用产生有趣的影响。实际上，标准基于梯度的优化方法在计算成本上显著低于现有的持续学习算法。本文中，我们研究了在数据重复出现的长任务序列中，使用基于梯度算法训练的深度神经网络渐进式知识积累（KA）过程。我们提出了一个新的框架SCoLe（持续学习缩放），用于探究知识积累，并发现灾难性遗忘对SGD训练的深度神经网络影响有限。当在数据稀疏重复出现的长序列上训练时，整体准确率反而提升——考虑到灾难性遗忘现象，这一结果可能违背直觉。我们通过实验探索了不同数据出现频率下深度神经网络中的知识积累，并提出了简单且可扩展的策略以增强深度神经网络中的知识积累。