With the success of pretraining techniques in representation learning, a number of continual learning methods based on pretrained models have been proposed. Some of these methods design continual learning mechanisms on the pre-trained representations and only allow minimum updates or even no updates of the backbone models during the training of continual learning. In this paper, we question whether the complexity of these models is needed to achieve good performance by comparing them to a simple baseline that we designed. We argue that the pretrained feature extractor itself can be strong enough to achieve a competitive or even better continual learning performance on Split-CIFAR100 and CoRe 50 benchmarks. To validate this, we conduct a very simple baseline that 1) use the frozen pretrained model to extract image features for every class encountered during the continual learning stage and compute their corresponding mean features on training data, and 2) predict the class of the input based on the nearest neighbor distance between test samples and mean features of the classes; i.e., Nearest Mean Classifier (NMC). This baseline is single-headed, exemplar-free, and can be task-free (by updating the means continually). This baseline achieved 88.53% on 10-Split-CIFAR-100, surpassing most state-of-the-art continual learning methods that are all initialized using the same pretrained transformer model. We hope our baseline may encourage future progress in designing learning systems that can continually add quality to the learning representations even if they started from some pretrained weights.

翻译：随着预训练技术在表示学习领域的成功，大量基于预训练模型的持续学习方法相继被提出。其中部分方法在预训练表示上设计持续学习机制，且在持续学习训练过程中仅允许对骨干网络进行最小化更新甚至完全不更新。本文通过将这类方法与我们所设计的简单基线进行比较，质疑其复杂性对于实现优异性能的必要性。我们认为，预训练特征提取器本身已足够强大，能够在Split-CIFAR100和CoRe50基准测试中取得具有竞争力甚至更优的持续学习表现。为验证这一观点，我们构建了一个极为简单的基线方案：1) 在持续学习阶段，使用冻结的预训练模型为每个遇到的类别提取图像特征，并在训练数据上计算各类别的平均特征；2) 根据测试样本与各类别平均特征的最近邻距离预测输入类别——即最近均值分类器（NMC）。该基线为单头模型、无需特征缓存，且可通过持续更新均值实现无任务依赖。此基线在10-Split-CIFAR-100上达到88.53%的准确率，超越了所有使用相同预训练Transformer模型初始化的现有持续学习方法。我们期望该基线能推动未来研究方向，即如何在预训练权重基础上持续提升学习表示的质量。