The recent success of large language models (LLMs) trained on static, pre-collected, general datasets has sparked numerous research directions and applications. One such direction addresses the non-trivial challenge of integrating pre-trained LLMs into dynamic data distributions, task structures, and user preferences. Pre-trained LLMs, when tailored for specific needs, often experience significant performance degradation in previous knowledge domains -- a phenomenon known as "catastrophic forgetting". While extensively studied in the continual learning (CL) community, it presents new manifestations in the realm of LLMs. In this survey, we provide a comprehensive overview of the current research progress on LLMs within the context of CL. This survey is structured into four main sections: we first describe an overview of continually learning LLMs, consisting of two directions of continuity: vertical continuity (or vertical continual learning), i.e., continual adaptation from general to specific capabilities, and horizontal continuity (or horizontal continual learning), i.e., continual adaptation across time and domains (Section 3). We then summarize three stages of learning LLMs in the context of modern CL: Continual Pre-Training (CPT), Domain-Adaptive Pre-training (DAP), and Continual Fine-Tuning (CFT) (Section 4). Then we provide an overview of evaluation protocols for continual learning with LLMs, along with the current available data sources (Section 5). Finally, we discuss intriguing questions pertaining to continual learning for LLMs (Section 6). The full list of papers examined in this survey is available at https://github.com/Wang-ML-Lab/llm-continual-learning-survey.

翻译：近期，基于静态、预收集的通用数据集训练的大语言模型取得了显著成功，催生了众多研究方向与应用。其中一个方向致力于解决将预训练大语言模型整合到动态数据分布、任务结构和用户偏好中的非平凡挑战。为特定需求定制的预训练大语言模型，往往在先前知识领域出现显著的性能下降——这一现象被称为“灾难性遗忘”。尽管持续学习领域对此已进行广泛研究，但该问题在大语言模型范畴内呈现出新的表现形式。本综述全面概述了当前在持续学习背景下大语言模型的研究进展。综述主体分为四个部分：首先，我们描述持续学习大语言模型的概览，包含两个连续性方向：垂直连续性（或称垂直持续学习），即从通用能力到特定能力的持续适应；以及水平连续性（或称水平持续学习），即跨时间和领域的持续适应（第3节）。随后，我们总结了现代持续学习背景下大语言模型学习的三个阶段：持续预训练、领域自适应预训练和持续微调（第4节）。接着，我们概述了用于大语言模型持续学习的评估协议，以及当前可用的数据资源（第5节）。最后，我们探讨了与大语言模型持续学习相关的若干引人深思的问题（第6节）。本综述所检阅的全部论文列表可在 https://github.com/Wang-ML-Lab/llm-continual-learning-survey 获取。