Nowadays, real-world applications often face streaming data, which requires the learning system to absorb new knowledge as data evolves. Continual Learning (CL) aims to achieve this goal and meanwhile overcome the catastrophic forgetting of former knowledge when learning new ones. Typical CL methods build the model from scratch to grow with incoming data. However, the advent of the pre-trained model (PTM) era has sparked immense research interest, particularly in leveraging PTMs' robust representational capabilities. This paper presents a comprehensive survey of the latest advancements in PTM-based CL. We categorize existing methodologies into three distinct groups, providing a comparative analysis of their similarities, differences, and respective advantages and disadvantages. Additionally, we offer an empirical study contrasting various state-of-the-art methods to highlight concerns regarding fairness in comparisons. The source code to reproduce these evaluations is available at: https://github.com/sun-hailong/LAMDA-PILOT

翻译：如今，现实应用常面临流式数据，这要求学习系统随着数据演变不断吸收新知识。持续学习旨在实现这一目标，同时克服学习新知识时对旧知识的灾难性遗忘。典型的持续学习方法从零开始构建模型以适配不断涌入的数据。然而，预训练模型时代的到来引发了广泛的研究兴趣，尤其是在利用预训练模型强大的表征能力方面。本文全面综述了基于预训练模型的持续学习的最新进展。我们将现有方法划分为三大类，对其相似性、差异以及各自优缺点进行了比较分析。此外，我们通过实证研究对比了多种最先进的方法，以凸显在公平性比较方面的关键问题。重现这些评估的源代码可在以下网址获取：https://github.com/sun-hailong/LAMDA-PILOT