The goal of continual learning (CL) is to retain knowledge across tasks, but this conflicts with strict privacy required for sensitive training data that prevents storing or memorising individual samples. This work explores the intersection of CL and differential privacy (DP). We advance the theoretical understanding and introduce methods for combining CL and DP. We formulate and clarify the theory for DP CL focusing on composition over tasks. We introduce different variants of choosing classifiers' output label space, show that choosing the output label space directly based on the task data is not DP, and offer a DP alternative. We propose a method for combining pre-trained models with DP prototype classifiers and parameter-efficient adapters learned under DP to address the trade-offs between privacy and utility in a CL setting. We also demonstrate the effectiveness of our methods for varying degrees of domain shift, for blurry tasks, and with different output label settings.

翻译：持续学习（CL）的目标是在不同任务间保留知识，但这与敏感训练数据所要求的严格隐私保护相冲突，后者禁止存储或记忆单个样本。本文探索了持续学习与差分隐私（DP）的交集。我们推进了理论理解，并提出了结合CL与DP的方法。我们针对DP CL建立了理论框架并加以澄清，重点关注跨任务的组合隐私。我们提出了选择分类器输出标签空间的不同变体，证明直接基于任务数据选择输出标签空间不符合DP要求，并提供了一个符合DP的替代方案。我们提出了一种方法，将预训练模型与DP原型分类器以及在DP下学习的参数高效适配器相结合，以解决CL设置中隐私与效用之间的权衡问题。我们还展示了所提方法在不同程度的域偏移、模糊任务以及不同输出标签设置下的有效性。