While federated learning is promising for privacy-preserving collaborative learning without revealing local data, it remains vulnerable to white-box attacks and struggles to adapt to heterogeneous clients. Federated distillation (FD), built upon knowledge distillation--an effective technique for transferring knowledge from a teacher model to student models--emerges as an alternative paradigm, which provides enhanced privacy guarantees and addresses model heterogeneity. Nevertheless, challenges arise due to variations in local data distributions and the absence of a well-trained teacher model, which leads to misleading and ambiguous knowledge sharing that significantly degrades model performance. To address these issues, this paper proposes a selective knowledge sharing mechanism for FD, termed Selective-FD. It includes client-side selectors and a server-side selector to accurately and precisely identify knowledge from local and ensemble predictions, respectively. Empirical studies, backed by theoretical insights, demonstrate that our approach enhances the generalization capabilities of the FD framework and consistently outperforms baseline methods. This study presents a promising direction for effective knowledge transfer in privacy-preserving collaborative learning.

翻译：尽管联邦学习因无需暴露本地数据即可实现隐私保护的协作学习而极具前景，但其仍易遭受白盒攻击，且难以适应异质性客户端。联邦蒸馏（Federated Distillation, FD）基于知识蒸馏这一将教师模型知识有效迁移至学生模型的技术，作为一种替代范式应运而生，它在提供更强隐私保障的同时兼顾了模型异质性。然而，由本地数据分布差异及缺乏训练完备的教师模型所引发的挑战，会导致误导性与模糊性知识共享，从而显著降低模型性能。针对上述问题，本文提出面向FD的选择性知识共享机制Selective-FD。该机制包含客户端选择器与服务器端选择器，可分别从本地预测与集成预测中精确、精准地识别有效知识。融合理论洞见的实证研究表明，本文方法增强了FD框架的泛化能力，且始终优于基线方法。本研究为隐私保护协作学习中实现高效知识迁移指明了一条富有前景的路径。