Online Knowledge Distillation (KD) is recently highlighted to train large models in Federated Learning (FL) environments. Many existing studies adopt the logit ensemble method to perform KD on the server side. However, they often assume that unlabeled data collected at the edge is centralized on the server. Moreover, the logit ensemble method personalizes local models, which can degrade the quality of soft targets, especially when data is highly non-IID. To address these critical limitations,we propose a novel on-device KD-based heterogeneous FL method. Our approach leverages a small auxiliary model to learn from labeled local data. Subsequently, a subset of clients with strong system resources transfers knowledge to a large model through on-device KD using their unlabeled data. Our extensive experiments demonstrate that our on-device KD-based heterogeneous FL method effectively utilizes the system resources of all edge devices as well as the unlabeled data, resulting in higher accuracy compared to SOTA KD-based FL methods.

翻译：在线知识蒸馏（KD）最近在联邦学习（FL）环境中训练大模型方面受到关注。许多现有研究采用对数集成方法在服务器端执行KD。然而，它们通常假设在边缘收集的未标记数据已集中到服务器上。此外，对数集成方法会个性化本地模型，这可能降低软目标的质量，尤其是在数据高度非独立同分布时。为了解决这些关键限制，我们提出了一种新颖的基于设备端KD的异构FL方法。我们的方法利用一个小的辅助模型从带标签的本地数据中学习。随后，一部分系统资源强的客户端使用其未标记数据，通过设备端KD将知识转移到一个大模型中。我们的大量实验表明，我们基于设备端KD的异构FL方法有效利用了所有边缘设备的系统资源以及未标记数据，与基于KD的SOTA FL方法相比，获得了更高的准确率。