Continual learning in Large Language Models (LLMs) is hindered by the plasticity-stability dilemma, where acquiring new capabilities often leads to catastrophic forgetting of previous knowledge. Existing methods typically treat parameters uniformly, failing to distinguish between specific task knowledge and shared capabilities. We introduce Mixture of Sparse Experts for Task-Agnostic Continual Learning, referred to as SETA, a framework that resolves the plasticity-stability conflict by decomposing the model into modular subspaces. Unlike standard updates, where tasks compete for the same parameters, SETA separates knowledge into unique experts, designed to isolate task-specific patterns, and shared experts, responsible for capturing common features. This structure is maintained through elastic weight anchoring, which protects critical shared knowledge and enables a unified gating network to automatically retrieve the correct expert combination for each task during inference. Extensive experiments across diverse domain-specific and general benchmarks demonstrate that SETA consistently outperforms state-of-the-art parameter-efficient fine-tuning-based continual learning methods.

翻译：大型语言模型（LLM）的持续学习受制于可塑性-稳定性困境：获取新能力常导致对已有知识的灾难性遗忘。现有方法通常对参数进行统一处理，未能区分特定任务知识与共享能力。本文提出面向任务无关持续学习的稀疏专家混合方法（简称SETA），该框架通过将模型分解为模块化子空间来解决可塑性-稳定性冲突。与任务竞争相同参数的标准更新方式不同，SETA将知识分离为两类专家：专为隔离任务特定模式而设计的独立专家，以及负责捕获共性特征的共享专家。该结构通过弹性权重锚定机制得以保持，该机制既保护关键共享知识，又使统一门控网络能在推理阶段自动检索适用于各任务的专家组合。在多样化领域专用及通用基准测试中的大量实验表明，SETA始终优于基于参数高效微调的先进持续学习方法。