Federated learning shows promise as a privacy-preserving collaborative learning technique. Existing heterogeneous federated learning mainly focuses on skewing the label distribution across clients. However, most approaches suffer from catastrophic forgetting and concept drift, mainly when the global distribution of all classes is extremely unbalanced and the data distribution of the client dynamically evolves over time. In this paper, we study the new task, i.e., Dynamic Heterogeneous Federated Learning (DHFL), which addresses the practical scenario where heterogeneous data distributions exist among different clients and dynamic tasks within the client. Accordingly, we propose a novel federated learning framework named Federated Multi-Level Prototypes (FedMLP) and design federated multi-level regularizations. To mitigate concept drift, we construct prototypes and semantic prototypes to provide fruitful generalization knowledge and ensure the continuity of prototype spaces. To maintain the model stability and consistency of convergence, three regularizations are introduced as training losses, i.e., prototype-based regularization, semantic prototype-based regularization, and federated inter-task regularization. Extensive experiments show that the proposed method achieves state-of-the-art performance in various settings.

翻译：联邦学习作为一种隐私保护的协作学习技术展现出巨大潜力。现有异构联邦学习主要聚焦于客户端间的标签分布偏移，然而多数方法在全局类别分布极度不平衡且客户端数据分布随时间动态演化时，易遭受灾难性遗忘和概念漂移问题。本文研究了新型任务——动态异构联邦学习（Dynamic Heterogeneous Federated Learning, DHFL），该任务解决了不同客户端间存在异构数据分布及客户端内部存在动态任务的实际场景。为此，我们提出名为联邦多层级原型（Federated Multi-Level Prototypes, FedMLP）的新型联邦学习框架，并设计了联邦多层级正则化方法。为缓解概念漂移，我们构建原型和语义原型，以提供丰富的泛化知识并确保原型空间的连续性。为维持模型稳定性与收敛一致性，引入三种正则化项作为训练损失：基于原型的正则化、基于语义原型的正则化以及联邦任务间正则化。大量实验表明，所提方法在多种设置下均取得了最先进的性能。