Federated learning is a specific distributed learning paradigm in which a central server aggregates updates from multiple clients' local models, thereby enabling the server to learn without requiring clients to upload their private data, maintaining data privacy. While existing federated learning methods are primarily designed for static data, real-world applications often require clients to learn new categories over time. This challenge necessitates the integration of continual learning techniques, resulting in federated continual learning (FCL). Although advanced prompt-based continual learning methods leverage pre-trained transformers to mitigate catastrophic forgetting, they do not adequately address the non-IID challenges in federated learning. To address both catastrophic forgetting and non-IID issues, we propose to use masked autoencoders (MAEs) as parameter-efficient federated continual learners, called pMAE. pMAE learns reconstructive prompt on the client side through image reconstruction using MAEs. On the server side, it reconstructs the uploaded restore information to capture the data distribution across previous tasks and different clients, using these reconstructed images to finetune discriminative prompt and classifier parameters designed for classification, thereby alleviating catastrophic forgetting and non-IID challenges on a global scale. Experimental results demonstrate that pMAE achieves performance comparable to existing prompt-based methods and can enhance their effectiveness, particularly when using self-supervised pre-trained transformers as the backbone. Code is available at: https://github.com/ycheoo/pMAE.

翻译：联邦学习是一种特定的分布式学习范式，其中中央服务器聚合来自多个客户端本地模型的更新，从而使服务器能够学习而无需客户端上传其私有数据，保持了数据隐私性。虽然现有的联邦学习方法主要针对静态数据设计，但现实应用通常要求客户端随时间学习新类别。这一挑战需要整合持续学习技术，从而形成联邦持续学习。尽管先进的基于提示的持续学习方法利用预训练的Transformer来缓解灾难性遗忘，但它们未能充分解决联邦学习中的非独立同分布挑战。为同时应对灾难性遗忘和非独立同分布问题，我们提出使用掩码自编码器作为参数高效的联邦持续学习器，称为pMAE。pMAE在客户端通过使用MAE进行图像重建来学习重建性提示。在服务器端，它重建上传的恢复信息以捕获跨先前任务和不同客户端的数据分布，并利用这些重建图像微调专为分类设计的判别性提示和分类器参数，从而在全局范围内缓解灾难性遗忘和非独立同分布挑战。实验结果表明，pMAE实现了与现有基于提示方法相当的性能，并能提升其有效性，特别是在使用自监督预训练Transformer作为骨干网络时。代码发布于：https://github.com/ycheoo/pMAE。