Current evaluations of Continual Learning (CL) methods typically assume that there is no constraint on training time and computation. This is an unrealistic assumption for any real-world setting, which motivates us to propose: a practical real-time evaluation of continual learning, in which the stream does not wait for the model to complete training before revealing the next data for predictions. To do this, we evaluate current CL methods with respect to their computational costs. We hypothesize that under this new evaluation paradigm, computationally demanding CL approaches may perform poorly on streams with a varying distribution. We conduct extensive experiments on CLOC, a large-scale dataset containing 39 million time-stamped images with geolocation labels. We show that a simple baseline outperforms state-of-the-art CL methods under this evaluation, questioning the applicability of existing methods in realistic settings. In addition, we explore various CL components commonly used in the literature, including memory sampling strategies and regularization approaches. We find that all considered methods fail to be competitive against our simple baseline. This surprisingly suggests that the majority of existing CL literature is tailored to a specific class of streams that is not practical. We hope that the evaluation we provide will be the first step towards a paradigm shift to consider the computational cost in the development of online continual learning methods.

翻译：当前对持续学习（CL）方法的评估通常假设训练时间和计算无约束。这一假设在现实场景中并不成立，因此我们提出：一种实用的持续学习实时评估方法，即数据流在模型完成训练前不会等待，而是直接揭示下一批预测数据。为此，我们评估了现有持续学习方法在计算成本方面的表现。我们假设，在这种新评估范式下，计算密集型的CL方法在分布多变的数据流中可能表现不佳。我们在CLOC（一个包含3900万张带地理标签时间戳图像的大规模数据集）上进行了广泛实验。结果表明，在该评估体系下，一个简单的基线方法超越了当前最先进的CL方法，质疑了现有方法在现实场景中的适用性。此外，我们探索了文献中常用的多种CL组件，包括记忆采样策略和正则化方法，发现所有被考虑的方法均无法与我们的简单基线竞争。这一发现令人惊讶地表明，现有CL文献中的大多数方法仅适用于特定类型的非实用数据流。我们希望本文提出的评估方法能成为推动在线持续学习方法开发中纳入计算成本考量的范式转变的第一步。