Continual Learning (CL) aims to sequentially train models on streams of incoming data that vary in distribution by preserving previous knowledge while adapting to new data. Current CL literature focuses on restricted access to previously seen data, while imposing no constraints on the computational budget for training. This is unreasonable for applications in-the-wild, where systems are primarily constrained by computational and time budgets, not storage. We revisit this problem with a large-scale benchmark and analyze the performance of traditional CL approaches in a compute-constrained setting, where effective memory samples used in training can be implicitly restricted as a consequence of limited computation. We conduct experiments evaluating various CL sampling strategies, distillation losses, and partial fine-tuning on two large-scale datasets, namely ImageNet2K and Continual Google Landmarks V2 in data incremental, class incremental, and time incremental settings. Through extensive experiments amounting to a total of over 1500 GPU-hours, we find that, under compute-constrained setting, traditional CL approaches, with no exception, fail to outperform a simple minimal baseline that samples uniformly from memory. Our conclusions are consistent in a different number of stream time steps, e.g., 20 to 200, and under several computational budgets. This suggests that most existing CL methods are particularly too computationally expensive for realistic budgeted deployment. Code for this project is available at: https://github.com/drimpossible/BudgetCL.

翻译：持续学习（CL）旨在通过保留先前知识的同时适应新数据，在分布变化的输入数据流上顺序训练模型。当前持续学习文献主要关注对先前见过数据的访问限制，而未对训练计算预算施加任何约束。这对于实际应用场景是不合理的——在现实系统中，主要受限于计算和时间预算而非存储容量。我们通过大规模基准测试重新审视该问题，分析传统持续学习方法在计算受限场景下的性能表现——此时训练中使用的有效记忆样本会因计算限制而被隐式约束。我们开展了广泛实验，评估了多种CL采样策略、蒸馏损失及部分微调方法在ImageNet2K和Continual Google Landmarks V2两个大规模数据集上的表现，涵盖数据增量、类别增量和时间增量三种设置。总计超过1500 GPU小时的实验结果表明：在计算受限条件下，所有传统持续学习方法无一例外地未能超越从记忆库中均匀采样的简单基线方法。该结论在20至200个不同的流时间步长及多种计算预算下均保持稳定。这表明现有大多数持续学习方法对于实际预算部署而言计算代价过高。项目代码详见：https://github.com/drimpossible/BudgetCL。