While federated learning is promising for privacy-preserving collaborative learning without revealing local data, it remains vulnerable to white-box attacks and struggles to adapt to heterogeneous clients. Federated distillation (FD), built upon knowledge distillation--an effective technique for transferring knowledge from a teacher model to student models--emerges as an alternative paradigm, which provides enhanced privacy guarantees and addresses model heterogeneity. Nevertheless, challenges arise due to variations in local data distributions and the absence of a well-trained teacher model, which leads to misleading and ambiguous knowledge sharing that significantly degrades model performance. To address these issues, this paper proposes a selective knowledge sharing mechanism for FD, termed Selective-FD. It includes client-side selectors and a server-side selector to accurately and precisely identify knowledge from local and ensemble predictions, respectively. Empirical studies, backed by theoretical insights, demonstrate that our approach enhances the generalization capabilities of the FD framework and consistently outperforms baseline methods. This study presents a promising direction for effective knowledge transfer in privacy-preserving collaborative learning.

翻译：联邦学习虽能在不泄露本地数据的前提下实现隐私保护的协作学习，但仍易受白盒攻击且难以适应异构客户端。联邦蒸馏（Federated Distillation, FD）基于知识蒸馏——一种将知识从教师模型有效迁移至学生模型的技术——作为替代范式应运而生，它能提供更强的隐私保障并解决模型异构性问题。然而，由于本地数据分布差异和缺乏训练良好的教师模型，会导致误导性与模糊的知识共享，显著降低模型性能。为解决上述问题，本文提出面向FD的选择性知识共享机制Selective-FD。该机制包含客户端选择器与服务器端选择器，可分别从本地预测与集成预测中精准识别有效知识。基于理论支撑的实证研究表明，本方法能增强FD框架的泛化能力，且始终优于基线方法。本研究为隐私保护协作学习中实现高效知识迁移提供了极具前景的方向。