Current evaluations of Continual Learning (CL) methods typically assume that there is no constraint on training time and computation. This is an unrealistic assumption for any real-world setting, which motivates us to propose: a practical real-time evaluation of continual learning, in which the stream does not wait for the model to complete training before revealing the next data for predictions. To do this, we evaluate current CL methods with respect to their computational costs. We conduct extensive experiments on CLOC, a large-scale dataset containing 39 million time-stamped images with geolocation labels. We show that a simple baseline outperforms state-of-the-art CL methods under this evaluation, questioning the applicability of existing methods in realistic settings. In addition, we explore various CL components commonly used in the literature, including memory sampling strategies and regularization approaches. We find that all considered methods fail to be competitive against our simple baseline. This surprisingly suggests that the majority of existing CL literature is tailored to a specific class of streams that is not practical. We hope that the evaluation we provide will be the first step towards a paradigm shift to consider the computational cost in the development of online continual learning methods.

翻译：当前对持续学习方法的评估通常假设训练时间和计算量不受限制。这对于任何真实场景而言都是不现实的假设，这促使我们提出：一种实用的持续学习实时评估方法，其中数据流在模型完成训练之前不会等待其进行下一项预测。为此，我们根据计算成本评估了当前持续学习方法。我们在CLOC（一个包含3900万带时间戳图像及地理标签的大规模数据集）上进行了广泛实验。结果表明，在此评估下，一个简单基线方法优于现有最先进的持续学习方法，这质疑了现有方法在现实场景中的适用性。此外，我们探索了文献中常用的各种持续学习组件，包括内存采样策略和正则化方法。我们发现所有被考虑的方法都无法与简单基线竞争。这一令人惊讶的结果表明，大多数现有持续学习文献是为特定类别的数据流量身定制的，而这并不实用。我们希望所提供的评估将成为迈向考虑计算成本的在线持续学习方法开发范式转变的第一步。