In Continual Learning (CL), using a Pre-Trained Model (PTM) as the feature extractor has become a popular practice. Accompanied by analytic classifiers, the PTM-based methods have achieved state-of-the-art performance in CL, in pursuit of the non-forgetting goal. Meanwhile, actively forgetting specific knowledge acquired during the CL phase is also essential in most service construction paradigms, for example, Mobile Crowd Sensing (MCS), where mobile edge nodes continuously collect sensory data and demand not only non-forgetting adaptation but also specific knowledge forgetting for privacy preservation. Thus, a unique problem, called Continual Unlearning (CU), arises when the forgetting requests show sequentially in CL. However, existing unlearning methods focus on single-shot joint forgetting and prove highly inadequate when applied to CU, including (1) violating the historical data privacy in CL and (2) vulnerably being overwhelmed or degraded with adversarially frequent requests. To handle the challenges of CU, we propose a gradient-free approach, called Analytic Continual Unlearning (ACU), for efficient and exact forgetting with historical data privacy preservation in PTM-based CL. In response to each unlearning request, our ACU recursively derives the analytical (i.e., closed-form) solutions via least squares in an interpretable manner. By meticulous design, our ACU is compatible with both sample-level and class-level unlearning requests. The theoretical and experimental evaluations validate our ACU's superiority in unlearning effectiveness, model fidelity, and system efficiency.

翻译：在持续学习（CL）中，使用预训练模型（PTM）作为特征提取器已成为一种流行做法。结合解析分类器，基于PTM的方法在CL中取得了最先进的性能，旨在实现非遗忘目标。与此同时，在大多数服务构建范式（例如移动群智感知（MCS））中，主动遗忘CL阶段获得的特定知识也至关重要。在该场景下，移动边缘节点持续收集传感数据，不仅需要非遗忘自适应，还需要为隐私保护而遗忘特定知识。因此，当遗忘请求在CL中顺序出现时，便产生了一个独特问题，即持续去学习（CU）。然而，现有的去学习方法专注于单次联合遗忘，在应用于CU时表现出明显不足，包括：（1）违反CL中的历史数据隐私；（2）在面对对抗性频繁请求时，容易因过载或性能退化而变得脆弱。为应对CU的挑战，我们提出了一种名为解析持续去学习（ACU）的无梯度方法，用于在基于PTM的CL中实现高效且精确的遗忘，同时保护历史数据隐私。针对每个去学习请求，我们的ACU以可解释的方式通过最小二乘法递归推导解析解（即闭式解）。通过精心设计，我们的ACU兼容样本级和类别级去学习请求。理论分析与实验评估验证了ACU在去学习效果、模型保真度和系统效率方面的优越性。