Synchronous federated learning (FL) is a popular paradigm for collaborative edge learning. It typically involves a set of heterogeneous devices locally training neural network (NN) models in parallel with periodic centralized aggregations. As some of the devices may have limited computational resources and varying availability, FL latency is highly sensitive to stragglers. Conventional approaches discard incomplete intra-model updates done by stragglers, alter the amount of local workload and architecture, or resort to asynchronous settings; which all affect the trained model performance under tight training latency constraints. In this work, we propose straggler-aware layer-wise federated learning (SALF) that leverages the optimization procedure of NNs via backpropagation to update the global model in a layer-wise fashion. SALF allows stragglers to synchronously convey partial gradients, having each layer of the global model be updated independently with a different contributing set of users. We provide a theoretical analysis, establishing convergence guarantees for the global model under mild assumptions on the distribution of the participating devices, revealing that SALF converges at the same asymptotic rate as FL with no timing limitations. This insight is matched with empirical observations, demonstrating the performance gains of SALF compared to alternative mechanisms mitigating the device heterogeneity gap in FL.

翻译：同步联邦学习（FL）是一种用于协作边缘学习的流行范式。它通常涉及一组异构设备并行训练神经网络（NN）模型，并定期进行集中聚合。由于部分设备可能计算资源有限且可用性动态变化，FL的延迟对拖延者高度敏感。传统方法会丢弃拖延者未完成的模型内部更新、调整本地工作负载与架构，或采用异步设置——这些策略在严格的训练延迟约束下均会影响训练模型的性能。本文提出感知拖延者的分层联邦学习（SALF），该方法利用神经网络通过反向传播的优化过程，以分层方式更新全局模型。SALF允许拖延者同步传递部分梯度，使全局模型的每一层能由不同的贡献用户子集独立更新。我们进行了理论分析，在参与设备分布的温和假设下建立了全局模型的收敛性保证，揭示了SALF与无时间限制的FL具有相同的渐近收敛速率。这一理论洞见与实验观察相吻合，证明了SALF相较于缓解FL设备异质性差距的替代机制的性能优势。