Federated Learning (FL) has been widely concerned for it enables decentralized learning while ensuring data privacy. However, most existing methods unrealistically assume that the classes encountered by local clients are fixed over time. After learning new classes, this assumption will make the model's catastrophic forgetting of old classes significantly severe. Moreover, due to the limitation of communication cost, it is challenging to use large-scale models in FL, which will affect the prediction accuracy. To address these challenges, we propose a novel framework, Federated Enhanced Transformer (FedET), which simultaneously achieves high accuracy and low communication cost. Specifically, FedET uses Enhancer, a tiny module, to absorb and communicate new knowledge, and applies pre-trained Transformers combined with different Enhancers to ensure high precision on various tasks. To address local forgetting caused by new classes of new tasks and global forgetting brought by non-i.i.d (non-independent and identically distributed) class imbalance across different local clients, we proposed an Enhancer distillation method to modify the imbalance between old and new knowledge and repair the non-i.i.d. problem. Experimental results demonstrate that FedET's average accuracy on representative benchmark datasets is 14.1% higher than the state-of-the-art method, while FedET saves 90% of the communication cost compared to the previous method.

翻译：摘要：联邦学习（FL）因其能够在保障数据隐私的同时实现去中心化学习而受到广泛关注。然而，现有大多数方法不切实际地假设本地客户端遇到的类别随时间保持不变。在学习新类别后，这一假设将导致模型对旧类别的灾难性遗忘显著加剧。此外，受通信成本限制，在FL中使用大规模模型十分困难，这会影响预测精度。为解决这些挑战，我们提出了一种新颖框架——联邦增强型Transformer（FedET），该框架同时实现了高精度与低通信成本。具体而言，FedET利用小型模块Enhancer吸收并传递新知识，并应用预训练的Transformer结合不同的Enhancer来确保各类任务的高精度。针对新任务的新类别导致的本地遗忘，以及不同本地客户端间非独立同分布（non-i.i.d.）类别不平衡带来的全局遗忘，我们提出了一种Enhancer蒸馏方法，以修正新旧知识间的不平衡并修复非独立同分布问题。实验结果表明，在代表性基准数据集上，FedET的平均准确率比现有最优方法高出14.1%，同时相较于先前方法节省了90%的通信成本。