Federated Learning (FL) has been successfully adopted for distributed training and inference of large-scale Deep Neural Networks (DNNs). However, DNNs are characterized by an extremely large number of parameters, thus, yielding significant challenges in exchanging these parameters among distributed nodes and managing the memory. Although recent DNN compression methods (e.g., sparsification, pruning) tackle such challenges, they do not holistically consider an adaptively controlled reduction of parameter exchange while maintaining high accuracy levels. We, therefore, contribute with a novel FL framework (coined FedDIP), which combines (i) dynamic model pruning with error feedback to eliminate redundant information exchange, which contributes to significant performance improvement, with (ii) incremental regularization that can achieve \textit{extreme} sparsity of models. We provide convergence analysis of FedDIP and report on a comprehensive performance and comparative assessment against state-of-the-art methods using benchmark data sets and DNN models. Our results showcase that FedDIP not only controls the model sparsity but efficiently achieves similar or better performance compared to other model pruning methods adopting incremental regularization during distributed model training. The code is available at: https://github.com/EricLoong/feddip.

翻译：联邦学习（FL）已成功应用于大规模深度神经网络（DNN）的分布式训练与推理。然而，DNN具有参数数量极其庞大的特点，这给分布式节点间的参数交换与内存管理带来了显著挑战。尽管近期DNN压缩方法（如稀疏化、剪枝）已应对此类挑战，但并未全面考虑在保持高精度水平的同时实现参数交换的自适应控制性缩减。为此，我们提出一种新型FL框架（称为FedDIP），该框架融合了：（i）结合误差反馈的动态模型剪枝，以消除冗余信息交换，从而显著提升性能；（ii）可实现模型极端稀疏化的增量正则化。我们提供了FedDIP的收敛性分析，并基于基准数据集与DNN模型，报告了与最先进方法的全面性能比较与评估。结果表明，FedDIP不仅能有效控制模型稀疏度，且在分布式模型训练中采用增量正则化时，其性能与其他模型剪枝方法相当或更优。代码开源地址：https://github.com/EricLoong/feddip。