In supervised continual learning, a deep neural network (DNN) is updated with an ever-growing data stream. Unlike the offline setting where data is shuffled, we cannot make any distributional assumptions about the data stream. Ideally, only one pass through the dataset is needed for computational efficiency. However, existing methods are inadequate and make many assumptions that cannot be made for real-world applications, while simultaneously failing to improve computational efficiency. In this paper, we propose a novel continual learning method, SIESTA based on wake/sleep framework for training, which is well aligned to the needs of on-device learning. The major goal of SIESTA is to advance compute efficient continual learning so that DNNs can be updated efficiently using far less time and energy. The principal innovations of SIESTA are: 1) rapid online updates using a rehearsal-free, backpropagation-free, and data-driven network update rule during its wake phase, and 2) expedited memory consolidation using a compute-restricted rehearsal policy during its sleep phase. For memory efficiency, SIESTA adapts latent rehearsal using memory indexing from REMIND. Compared to REMIND and prior arts, SIESTA is far more computationally efficient, enabling continual learning on ImageNet-1K in under 2 hours on a single GPU; moreover, in the augmentation-free setting it matches the performance of the offline learner, a milestone critical to driving adoption of continual learning in real-world applications.

翻译：在监督持续学习中，深度神经网络（DNN）需不断更新持续增长的数据流。与数据可随机打乱的离线场景不同，我们无法对数据流的分布性质做出任何假设。理想情况下，仅需对数据集进行一次遍历即可实现计算效率。然而，现有方法存在不足：它们为实际应用场景做出诸多不可实现的假设，同时未能有效提升计算效率。本文提出一种基于唤醒/睡眠训练框架的新型持续学习方法SIESTA，该方法与设备端学习需求高度契合。SIESTA的核心目标是推动计算高效型持续学习发展，使DNN能够用更少的时间和能量实现高效更新。其主要创新包括：1）在唤醒阶段通过无重放、无反向传播且数据驱动的网络更新规则实现快速在线更新；2）在睡眠阶段采用计算受限的重放策略加速记忆巩固。为提升内存效率，SIESTA采用源自REMIND的内存索引机制进行隐式重放。与REMIND及先前方法相比，SIESTA具有显著更高的计算效率——可在单GPU上于2小时内完成ImageNet-1K数据集的持续学习；此外，在无数据增强场景下，其性能与离线学习者相当，这一里程碑式成果对推动持续学习在实际应用中的普及至关重要。