Current evaluations of Continual Learning (CL) methods typically assume that there is no constraint on training time and computation. This is an unrealistic assumption for any real-world setting, which motivates us to propose: a practical real-time evaluation of continual learning, in which the stream does not wait for the model to complete training before revealing the next data for predictions. To do this, we evaluate current CL methods with respect to their computational costs. We conduct extensive experiments on CLOC, a large-scale dataset containing 39 million time-stamped images with geolocation labels. We show that a simple baseline outperforms state-of-the-art CL methods under this evaluation, questioning the applicability of existing methods in realistic settings. In addition, we explore various CL components commonly used in the literature, including memory sampling strategies and regularization approaches. We find that all considered methods fail to be competitive against our simple baseline. This surprisingly suggests that the majority of existing CL literature is tailored to a specific class of streams that is not practical. We hope that the evaluation we provide will be the first step towards a paradigm shift to consider the computational cost in the development of online continual learning methods.

翻译：目前对持续学习（CL）方法的评估通常假定训练时间和计算资源无限制。这一假设在真实场景中并不成立，因此我们提出一种实用的实时评估框架，要求模型在数据流不等待训练完成的情况下持续进行预测。为此，我们从计算成本角度评估现有CL方法。基于包含3900万张带地理标签的时间戳图像的大规模数据集CLOC进行广泛实验后，我们发现一个简单基线方法在该评估体系下超越了当前最先进的CL方法，这质疑了现有方法在实际场景中的适用性。此外，我们探究了文献中常用的各类CL组件（包括记忆采样策略与正则化方法），发现所有被考察方法均无法与简单基线竞争。这一结果惊人地表明，现有大多数CL研究仅适用于特定类型的非实用数据流。我们希望所提出的评估框架能成为推动在线持续学习方法转向关注计算成本的重要第一步。