Federated Learning (FL) is a distributed and privacy-preserving machine learning paradigm that coordinates multiple clients to train a model while keeping the raw data localized. However, this traditional FL poses some challenges, including privacy risks, data heterogeneity, communication bottlenecks, and system heterogeneity issues. To tackle these challenges, knowledge distillation (KD) has been widely applied in FL since 2020. KD is a validated and efficacious model compression and enhancement algorithm. The core concept of KD involves facilitating knowledge transfer between models by exchanging logits at intermediate or output layers. These properties make KD an excellent solution for the long-lasting challenges in FL. Up to now, there have been few reviews that summarize and analyze the current trend and methods for how KD can be applied in FL efficiently. This article aims to provide a comprehensive survey of KD-based FL, focusing on addressing the above challenges. First, we provide an overview of KD-based FL, including its motivation, basics, taxonomy, and a comparison with traditional FL and where KD should execute. We also analyze the critical factors in KD-based FL in the appendix, including teachers, knowledge, data, and methods. We discuss how KD can address the challenges in FL, including privacy protection, data heterogeneity, communication efficiency, and personalization. Finally, we discuss the challenges facing KD-based FL algorithms and future research directions. We hope this survey can provide insights and guidance for researchers and practitioners in the FL area.

翻译：联邦学习（Federated Learning, FL）是一种分布式且保护隐私的机器学习范式，它协调多个客户端协同训练模型，同时保持原始数据本地化。然而，这种传统的联邦学习带来了一些挑战，包括隐私风险、数据异质性、通信瓶颈以及系统异质性问题。为应对这些挑战，知识蒸馏（Knowledge Distillation, KD）自2020年起被广泛应用于联邦学习中。KD是一种经过验证且高效的模型压缩与增强算法。其核心概念在于通过交换中间层或输出层的逻辑值（logits）来促进模型间的知识迁移。这些特性使得KD成为解决联邦学习中长期存在挑战的优秀方案。迄今为止，少有综述性工作系统总结与分析当前如何将KD高效应用于联邦学习的趋势与方法。本文旨在对基于KD的联邦学习进行全面综述，重点关注如何应对上述挑战。首先，我们概述基于KD的联邦学习，包括其动机、基础、分类体系，并与传统联邦学习进行比较，同时探讨KD应在何处执行。附录中我们还分析了基于KD的联邦学习中的关键因素，包括教师模型、知识类型、数据及方法。我们讨论了KD如何应对联邦学习中的挑战，包括隐私保护、数据异质性、通信效率与个性化。最后，我们探讨了基于KD的联邦学习算法面临的挑战及未来研究方向。我们希望本综述能为联邦学习领域的研究者与实践者提供见解与指导。