Efficiently running federated learning (FL) on resource-constrained devices is challenging since they are required to train computationally intensive deep neural networks (DNN) independently. DNN partitioning-based FL (DPFL) has been proposed as one mechanism to accelerate training where the layers of a DNN (or computation) are offloaded from the device to the server. However, this creates significant communication overheads since the intermediate activation and gradient need to be transferred between the device and the server during training. While current research reduces the communication introduced by DNN partitioning using local loss-based methods, we demonstrate that these methods are ineffective in improving the overall efficiency (communication overhead and training speed) of a DPFL system. This is because they suffer from accuracy degradation and ignore the communication costs incurred when transferring the activation from the device to the server. This article proposes EcoFed - a communication efficient framework for DPFL systems. EcoFed eliminates the transmission of the gradient by developing pre-trained initialization of the DNN model on the device for the first time. This reduces the accuracy degradation seen in local loss-based methods. In addition, EcoFed proposes a novel replay buffer mechanism and implements a quantization-based compression technique to reduce the transmission of the activation. It is experimentally demonstrated that EcoFed can reduce the communication cost by up to 133x and accelerate training by up to 21x when compared to classic FL. Compared to vanilla DPFL, EcoFed achieves a 16x communication reduction and 2.86x training time speed-up. EcoFed is available from https://github.com/blessonvar/EcoFed.

翻译：在资源受限设备上高效运行联邦学习颇具挑战，因为设备需独立训练计算密集的深度神经网络（DNN）。基于DNN分区的联邦学习（DPFL）作为一种加速训练机制被提出，其将DNN层级（或计算任务）从设备卸载至服务器。然而，这产生了显著通信开销，因为训练过程中设备与服务器间需传输中间激活值与梯度。尽管现有研究采用基于局部损失的方法降低DNN分区引入的通信量，但实验表明，这些方法无法有效提升DPFL系统的整体效率（通信开销与训练速度）。原因在于此类方法存在精度退化问题，且忽略了设备向服务器传输激活值所产生的通信成本。本文提出EcoFed——面向DPFL系统的高效通信框架。EcoFed首次通过开发设备端DNN模型的预训练初始化机制消除了梯度传输，从而缓解了基于局部损失方法中的精度退化问题。此外，EcoFed提出新颖的重放缓冲区机制，并采用基于量化的压缩技术减少激活值传输。实验证明，相比经典联邦学习，EcoFed可将通信成本降低高达133倍，训练速度提升21倍；与原始DPFL相比，EcoFed实现16倍通信缩减与2.86倍训练时间加速。EcoFed开源代码见https://github.com/blessonvar/EcoFed。