Federated Learning (FL) is an innovative distributed machine learning paradigm that enables neural network training across devices without centralizing data. While this addresses issues of information sharing and data privacy, challenges arise from data heterogeneity across clients and increasing network scale, leading to impacts on model performance and training efficiency. Previous research shows that in IID environments, the parameter structure of the model is expected to adhere to certain specific consistency principles. Thus, identifying and regularizing these consistencies can mitigate issues from heterogeneous data. We found that both soft labels derived from knowledge distillation and the classifier head parameter matrix, when multiplied by their own transpose, capture the intrinsic relationships between data classes. These shared relationships suggest inherent consistency. Therefore, the work in this paper identifies the consistency between the two and leverages it to regulate training, underpinning our proposed FedDW framework. Experimental results show FedDW outperforms 10 state-of-the-art FL methods, improving accuracy by an average of 3% in highly heterogeneous settings. Additionally, we provide a theoretical proof that FedDW offers higher efficiency, with the additional computational load from backpropagation being negligible. The code is available at https://github.com/liuvvvvv1/FedDW.

翻译：联邦学习（Federated Learning, FL）是一种创新的分布式机器学习范式，它使得神经网络能够在设备间进行训练而无需集中数据。尽管这解决了信息共享和数据隐私问题，但来自客户端的数据异构性以及不断增长的网络规模带来了挑战，影响了模型性能和训练效率。先前研究表明，在独立同分布（IID）环境中，模型的参数结构预期遵循某些特定的一致性原理。因此，识别并正则化这些一致性可以缓解异构数据带来的问题。我们发现，从知识蒸馏中得到的软标签以及分类器头参数矩阵与其自身转置相乘时，均能捕捉数据类别间的内在关系。这些共享关系暗示了内在的一致性。因此，本文的工作识别了两者之间的一致性，并利用它来调控训练，从而支撑了我们提出的FedDW框架。实验结果表明，FedDW在高度异构设置下优于10种先进的FL方法，平均准确率提升了3%。此外，我们提供了理论证明，表明FedDW具有更高的效率，反向传播带来的额外计算负载可以忽略不计。代码可在 https://github.com/liuvvvvv1/FedDW 获取。