Supervised Continual learning involves updating a deep neural network (DNN) from an ever-growing stream of labeled data. While most work has focused on overcoming catastrophic forgetting, one of the major motivations behind continual learning is being able to efficiently update a network with new information, rather than retraining from scratch on the training dataset as it grows over time. Despite recent continual learning methods largely solving the catastrophic forgetting problem, there has been little attention paid to the efficiency of these algorithms. Here, we study recent methods for incremental class learning and illustrate that many are highly inefficient in terms of compute, memory, and storage. Some methods even require more compute than training from scratch! We argue that for continual learning to have real-world applicability, the research community cannot ignore the resources used by these algorithms. There is more to continual learning than mitigating catastrophic forgetting.

翻译：监督式持续学习涉及从不断增长的标记数据流中更新深度神经网络（DNN）。尽管多数研究聚焦于克服灾难性遗忘，但持续学习的主要动机之一在于能够高效地用新信息更新网络，而非随着时间推移在训练数据集上从头重新训练。尽管近期的持续学习方法在很大程度上解决了灾难性遗忘问题，但这些算法的效率却鲜受关注。本文研究了增量式类别学习的最新方法，并表明许多方法在计算、内存和存储方面效率极低。某些方法甚至需要比从头训练更多的计算量！我们认为，若要使持续学习具备实际应用价值，研究界不能忽视这些算法所消耗的资源。持续学习的内涵远不止缓解灾难性遗忘。