Federated Learning (FL) is a promising technique for the collaborative training of deep neural networks across multiple devices while preserving data privacy. Despite its potential benefits, FL is hindered by excessive communication costs due to repeated server-client communication during training. To address this challenge, model compression techniques, such as sparsification and weight clustering are applied, which often require modifying the underlying model aggregation schemes or involve cumbersome hyperparameter tuning, with the latter not only adjusts the model's compression rate but also limits model's potential for continuous improvement over growing data. In this paper, we propose FedCompress, a novel approach that combines dynamic weight clustering and server-side knowledge distillation to reduce communication costs while learning highly generalizable models. Through a comprehensive evaluation on diverse public datasets, we demonstrate the efficacy of our approach compared to baselines in terms of communication costs and inference speed. We will make our implementation public upon acceptance.

翻译：联邦学习（FL）是一种跨多个设备协作训练深度神经网络的同时保护数据隐私的有效技术。尽管具有潜在优势，但FL因训练过程中服务器与客户端间频繁通信导致的过高通信开销而受到制约。为解决这一挑战，现有研究采用稀疏化和权值聚类等模型压缩技术，但这些方法往往需要修改底层模型聚合方案或涉及繁琐的超参数调优——后者不仅调节模型压缩率，还会限制模型随数据增长实现持续优化的潜力。本文提出FedCompress，一种融合动态权值聚类与服务器端知识蒸馏的新型方法，旨在降低通信开销的同时学习强泛化能力的模型。通过在不同公开数据集上的全面评估，我们展示了该方法在通信开销与推理速度方面相较于基线的有效性。论文接收后，我们将公开代码实现。