Federated learning is a distributed machine learning paradigm designed to protect data privacy. However, data heterogeneity across various clients results in catastrophic forgetting, where the model rapidly forgets previous knowledge while acquiring new knowledge. To address this challenge, personalized federated learning has emerged to customize a personalized model for each client. However, the inherent limitation of this mechanism is its excessive focus on personalization, potentially hindering the generalization of those models. In this paper, we present a novel personalized federated learning method that uses global and historical models as teachers and the local model as the student to facilitate comprehensive knowledge distillation. The historical model represents the local model from the last round of client training, containing historical personalized knowledge, while the global model represents the aggregated model from the last round of server aggregation, containing global generalized knowledge. By applying knowledge distillation, we effectively transfer global generalized knowledge and historical personalized knowledge to the local model, thus mitigating catastrophic forgetting and enhancing the general performance of personalized models. Extensive experimental results demonstrate the significant advantages of our method.

翻译：联邦学习是一种旨在保护数据隐私的分布式机器学习范式。然而，不同客户端间的数据异质性会导致灾难性遗忘问题，即模型在获取新知识的同时迅速遗忘先前学到的知识。为应对这一挑战，个性化联邦学习应运而生，旨在为每个客户端定制个性化模型。然而，该机制固有的局限性在于其过度聚焦于个性化，可能阻碍模型的泛化能力。本文提出一种新颖的个性化联邦学习方法，以全局模型和历史模型作为教师模型，本地模型作为学生模型，实现全面知识蒸馏。历史模型代表上一轮客户端训练得到的本地模型，蕴含历史个性化知识；全局模型则代表上一轮服务器聚合产生的聚合模型，包含全局泛化知识。通过知识蒸馏技术，我们将全局泛化知识与历史个性化知识有效迁移至本地模型，从而缓解灾难性遗忘问题，并提升个性化模型的综合性能。大量实验结果验证了本方法的显著优势。