Federated Learning (FL) has emerged as a prominent alternative to the traditional centralized learning approach. Generally speaking, FL is a decentralized approach that allows for collaborative training of Machine Learning (ML) models across multiple local nodes, ensuring data privacy and security while leveraging diverse datasets. Conventional FL, however, is susceptible to gradient inversion attacks, restrictively enforces a uniform architecture on local models, and suffers from model heterogeneity (model drift) due to non-IID local datasets. To mitigate some of these challenges, the new paradigm of Federated Knowledge Distillation (FKD) has emerged. FDK is developed based on the concept of Knowledge Distillation (KD), which involves extraction and transfer of a large and well-trained teacher model's knowledge to lightweight student models. FKD, however, still faces the model drift issue. Intuitively speaking, not all knowledge is universally beneficial due to the inherent diversity of data among local nodes. This calls for innovative mechanisms to evaluate the relevance and effectiveness of each client's knowledge for others, to prevent propagation of adverse knowledge. In this context, the paper proposes Effective Knowledge Fusion (KnFu) algorithm that evaluates knowledge of local models to only fuse semantic neighbors' effective knowledge for each client. The KnFu is a personalized effective knowledge fusion scheme for each client, that analyzes effectiveness of different local models' knowledge prior to the aggregation phase. Comprehensive experiments were performed on MNIST and CIFAR10 datasets illustrating effectiveness of the proposed KnFu in comparison to its state-of-the-art counterparts. A key conclusion of the work is that in scenarios with large and highly heterogeneous local datasets, local training could be preferable to knowledge fusion-based solutions.

翻译：联邦学习（Federated Learning, FL）已成为传统集中式学习的重要替代方案。一般而言，FL是一种分布式方法，允许在多个本地节点上协同训练机器学习（ML）模型，在利用多样化数据集的同时确保数据隐私与安全。然而，传统FL易受梯度反转攻击，强制要求本地模型采用统一架构，并因非独立同分布（non-IID）本地数据集而面临模型异构性（模型漂移）问题。为缓解部分挑战，新范式联邦知识蒸馏（Federated Knowledge Distillation, FKD）应运而生。FKD基于知识蒸馏（Knowledge Distillation, KD）概念开发，涉及将大型且训练良好的教师模型知识提取并迁移至轻量级学生模型。但FKD仍面临模型漂移问题。直观而言，由于本地节点间数据固有的多样性，并非所有知识都具有普遍益处。这需要创新机制来评估每个客户端知识对其他客户端的相关性与有效性，以防止有害知识的传播。在此背景下，本文提出高效知识融合（Effective Knowledge Fusion, KnFu）算法，该算法评估本地模型知识，仅融合每个客户端的语义邻居的有效知识。KnFu是一种针对每个客户端的个性化高效知识融合方案，在聚合阶段前分析不同本地模型知识的有效性。在MNIST和CIFAR10数据集上进行的全面实验表明，与最先进方法相比，所提出的KnFu具有有效性。本研究的关键结论是：在本地数据集规模较大且高度异构的场景下，本地训练可能优于基于知识融合的解决方案。