The recent success of large language models (LLMs) trained on static, pre-collected, general datasets has sparked numerous research directions and applications. One such direction addresses the non-trivial challenge of integrating pre-trained LLMs into dynamic data distributions, task structures, and user preferences. Pre-trained LLMs, when tailored for specific needs, often experience significant performance degradation in previous knowledge domains -- a phenomenon known as "catastrophic forgetting". While extensively studied in the continual learning (CL) community, it presents new manifestations in the realm of LLMs. In this survey, we provide a comprehensive overview of the current research progress on LLMs within the context of CL. This survey is structured into four main sections: we first describe an overview of continually learning LLMs, consisting of two directions of continuity: vertical continuity (or vertical continual learning), i.e., continual adaptation from general to specific capabilities, and horizontal continuity (or horizontal continual learning), i.e., continual adaptation across time and domains (Section 3). We then summarize three stages of learning LLMs in the context of modern CL: Continual Pre-Training (CPT), Domain-Adaptive Pre-training (DAP), and Continual Fine-Tuning (CFT) (Section 4). Then we provide an overview of evaluation protocols for continual learning with LLMs, along with the current available data sources (Section 5). Finally, we discuss intriguing questions pertaining to continual learning for LLMs (Section 6). The full list of papers examined in this survey is available at https://github.com/Wang-ML-Lab/llm-continual-learning-survey.

翻译：近年来，基于静态预收集通用数据集训练的大型语言模型（LLMs）所取得的成功，催生了众多研究方向和应用。其中一个方向应对将预训练LLMs整合到动态数据分布、任务结构和用户偏好中的非平凡挑战。针对特定需求进行适配的预训练LLMs，往往会在先前知识领域出现显著的性能下降——这一现象被称为“灾难性遗忘”。尽管持续学习领域对此已有广泛研究，但在LLMs领域却呈现出新的表现形式。本综述系统梳理了当前LLMs在持续学习背景下的研究进展，共分为四个主要部分：首先概述持续学习LLMs的总体框架，包含两类连续性——垂直连续性（或称垂直持续学习），即从通用能力到专用能力的持续适配；水平连续性（或称水平持续学习），即跨时间和跨领域的持续适配（第3节）。随后总结现代持续学习背景下LLMs的三个学习阶段：持续预训练（CPT）、领域自适应预训练（DAP）和持续微调（CFT）（第4节）。接着介绍LLMs持续学习的评估协议及现有数据资源（第5节）。最后探讨LLMs持续学习领域的若干关键问题（第6节）。本综述涉及的全部论文列表请参见：https://github.com/Wang-ML-Lab/llm-continual-learning-survey。