Incremental learning is the ability of systems to acquire knowledge over time, enabling their adaptation and generalization to novel tasks. It is a critical ability for intelligent, real-world systems, especially when data changes frequently or is limited. This review provides a comprehensive analysis of incremental learning in Large Language Models. It synthesizes the state-of-the-art incremental learning paradigms, including continual learning, meta-learning, parameter-efficient learning, and mixture-of-experts learning. We demonstrate their utility for incremental learning by describing specific achievements from these related topics and their critical factors. An important finding is that many of these approaches do not update the core model, and none of them update incrementally in real-time. The paper highlights current problems and challenges for future research in the field. By consolidating the latest relevant research developments, this review offers a comprehensive understanding of incremental learning and its implications for designing and developing LLM-based learning systems.

翻译：增量学习是系统随时间获取知识的能力，使其能够适应并泛化至新任务。对于智能化的现实世界系统而言，这是一项关键能力，尤其是在数据频繁变化或有限的情况下。本综述全面分析了大型语言模型中的增量学习。它综合了当前最先进的增量学习范式，包括持续学习、元学习、参数高效学习和混合专家学习。我们通过描述这些相关主题的具体成果及其关键因素，展示了它们在增量学习中的效用。一个重要的发现是，许多此类方法并未更新核心模型，且没有一种方法能实时进行增量更新。本文指出了当前存在的问题及未来研究面临的挑战。通过整合最新的相关研究进展，本综述提供了对增量学习及其对设计和开发基于LLM的学习系统之影响的全面理解。