In this paper, we focus on the important yet understudied problem of Continual Federated Learning (CFL), where a server communicates with a set of clients to incrementally learn new concepts over time without sharing or storing any data. The complexity of this problem is compounded by challenges from both the Continual and Federated Learning perspectives. Specifically, models trained in a CFL setup suffer from catastrophic forgetting which is exacerbated by data heterogeneity across clients. Existing attempts at this problem tend to impose large overheads on clients and communication channels or require access to stored data which renders them unsuitable for real-world use due to privacy. In this paper, we attempt to tackle forgetting and heterogeneity while minimizing overhead costs and without requiring access to any stored data. We achieve this by leveraging a prompting based approach (such that only prompts and classifier heads have to be communicated) and proposing a novel and lightweight generation and distillation scheme to consolidate client models at the server. We formulate this problem for image classification and establish strong baselines for comparison, conduct experiments on CIFAR-100 as well as challenging, large-scale datasets like ImageNet-R and DomainNet. Our approach outperforms both existing methods and our own baselines by as much as 7% while significantly reducing communication and client-level computation costs.

翻译：本文聚焦于持续联邦学习（Continual Federated Learning, CFL）这一重要但尚未充分研究的问题——在服务器与一组客户端通信的过程中，无需共享或存储任何数据，即可逐步学习新概念。该问题的复杂性源于持续学习与联邦学习两大视角带来的挑战：在CFL设定下训练的模型易遭受灾难性遗忘，且该问题因客户端间的数据异质性而加剧。现有针对该问题的尝试往往给客户端和通信信道带来较大开销，或需要访问存储数据，因其隐私问题而无法适用于实际场景。本文致力于在最小化开销且无需访问任何存储数据的条件下，解决遗忘与异质性挑战。我们通过以下方式实现目标：基于提示的方法（使得仅需传输提示与分类器头部），并提出一种新颖的轻量级生成与蒸馏方案，用于在服务器端整合客户端模型。我们针对图像分类任务对该问题进行建模，建立了强基线用于对比，并在CIFAR-100以及ImageNet-R、DomainNet等大规模挑战性数据集上开展实验。本方法在显著降低通信与客户端计算开销的同时，相较于现有方法与自建基线，性能提升高达7%。