Neural networks are very effective when trained on large datasets for a large number of iterations. However, when they are trained on non-stationary streams of data and in an online fashion, their performance is reduced (1) by the online setup, which limits the availability of data, (2) due to catastrophic forgetting because of the non-stationary nature of the data. Furthermore, several recent works (Caccia et al., 2022; Lange et al., 2023) arXiv:2205.1345(2) showed that replay methods used in continual learning suffer from the stability gap, encountered when evaluating the model continually (rather than only on task boundaries). In this article, we study the effect of model ensembling as a way to improve performance and stability in online continual learning. We notice that naively ensembling models coming from a variety of training tasks increases the performance in online continual learning considerably. Starting from this observation, and drawing inspirations from semi-supervised learning ensembling methods, we use a lightweight temporal ensemble that computes the exponential moving average of the weights (EMA) at test time, and show that it can drastically increase the performance and stability when used in combination with several methods from the literature.

翻译：神经网络在大型数据集上经过大量迭代训练时表现十分有效。然而，当它们在非平稳数据流上以在线方式训练时，其性能会因以下因素而降低：（1）在线设置限制了数据的可用性；（2）数据非平稳特性导致的灾难性遗忘。此外，近期多项研究（Caccia 等，2022；Lange 等，2023）arXiv:2205.1345(2) 表明，持续学习中使用的重放方法存在稳定性缺口，这种缺口在持续评估模型（而非仅在任务边界评估）时尤为明显。本文研究了模型集成作为提升在线持续学习性能与稳定性的方法。我们发现，对来自不同训练任务的模型进行朴素集成能显著提升在线持续学习的性能。基于此观察，并借鉴半监督学习集成方法，我们采用一种轻量级的时间集成方法，在测试时计算权重的指数移动平均（EMA）。实验表明，该方法与文献中的多种方法结合使用时，能大幅提升性能与稳定性。