Most existing federated learning algorithms are based on the vanilla FedAvg scheme. However, with the increase of data complexity and the number of model parameters, the amount of communication traffic and the number of iteration rounds for training such algorithms increases significantly, especially in non-independently and homogeneously distributed scenarios, where they do not achieve satisfactory performance. In this work, we propose FedND: federated learning with noise distillation. The main idea is to use knowledge distillation to optimize the model training process. In the client, we propose a self-distillation method to train the local model. In the server, we generate noisy samples for each client and use them to distill other clients. Finally, the global model is obtained by the aggregation of local models. Experimental results show that the algorithm achieves the best performance and is more communication-efficient than state-of-the-art methods.

翻译：大多数现有的联邦学习算法基于原始FedAvg方案。然而，随着数据复杂性和模型参数数量的增加，此类算法的通信流量和训练迭代轮次显著上升，尤其在非独立同分布场景下无法达到令人满意的性能。本文提出FedND：基于噪声蒸馏的联邦学习。其核心思想是利用知识蒸馏优化模型训练过程。在客户端层面，我们提出一种自蒸馏方法训练本地模型；在服务器端，我们为每个客户端生成噪声样本，并利用这些样本蒸馏其他客户端。最终通过聚合各局部模型得到全局模型。实验结果表明，该算法实现了最优性能，且相比现有最先进方法具有更高的通信效率。