Nowadays, real-world applications often face streaming data, which requires the learning system to absorb new knowledge as data evolves. Continual Learning (CL) aims to achieve this goal and meanwhile overcome the catastrophic forgetting of former knowledge when learning new ones. Typical CL methods build the model from scratch to grow with incoming data. However, the advent of the pre-trained model (PTM) era has sparked immense research interest, particularly in leveraging PTMs' robust representational capabilities. This paper presents a comprehensive survey of the latest advancements in PTM-based CL. We categorize existing methodologies into three distinct groups, providing a comparative analysis of their similarities, differences, and respective advantages and disadvantages. Additionally, we offer an empirical study contrasting various state-of-the-art methods to highlight concerns regarding fairness in comparisons. The source code to reproduce these evaluations is available at: https://github.com/sun-hailong/LAMDA-PILOT

翻译：如今，现实世界应用常面临流式数据，这要求学习系统在数据演化过程中吸收新知识。持续学习旨在实现这一目标，同时在学习新知识时克服对旧知识的灾难性遗忘。典型的持续学习方法从零开始构建模型以适应不断到来的数据。然而，预训练模型时代的到来激发了广泛的研究兴趣，特别是在利用PTM强大的表征能力方面。本文全面综述了基于PTM的持续学习最新进展。我们将现有方法划分为三个不同类别，比较分析其相似性、差异性及各自优缺点。此外，我们还通过实证研究对比了多种最先进方法，揭示了比较公平性方面的关注问题。复现这些评估的源代码可访问：https://github.com/sun-hailong/LAMDA-PILOT