Federated Learning (FL) is the state-of-the-art approach for learning from decentralized data in privacy-constrained scenarios. As the current literature reports, the main problems associated with FL refer to system and statistical challenges: the former ones demand for efficient learning from edge devices, including lowering communication bandwidth and frequency, while the latter require algorithms robust to non-iidness. State-of-art approaches either guarantee convergence at increased communication cost or are not sufficiently robust to handle extreme heterogeneous local distributions. In this work we propose a novel generalization of the heavy-ball momentum, and present FedHBM to effectively address statistical heterogeneity in FL without introducing any communication overhead. We conduct extensive experimentation on common FL vision and NLP datasets, showing that our FedHBM algorithm empirically yields better model quality and higher convergence speed w.r.t. the state-of-art, especially in pathological non-iid scenarios. While being designed for cross-silo settings, we show how FedHBM is applicable in moderate-to-high cross-device scenarios, and how good model initializations (e.g. pre-training) can be exploited for prompt acceleration. Extended experimentation on large-scale real-world federated datasets further corroborates the effectiveness of our approach for real-world FL applications.

翻译：联邦学习（FL）是在隐私约束场景下从分散数据中学习的前沿方法。现有文献报告指出，FL的主要问题涉及系统与统计挑战：前者要求实现边缘设备的高效学习，包括降低通信带宽与频率；后者则需要算法对非独立同分布（non-iid）数据具有鲁棒性。当前先进方法要么在增加通信成本的前提下保证收敛性，要么无法充分应对极端异构的本地分布。本文提出一种重球动量的新颖广义化方法，并引入FedHBM，在不增加任何通信开销的前提下有效解决FL中的统计异构性问题。我们在常见的FL视觉与自然语言处理（NLP）数据集上开展大量实验，结果表明：相较于现有先进方法，FedHBM算法在经验上能实现更优的模型质量与更快的收敛速度，尤其在病态非独立同分布场景中效果显著。尽管该算法专为跨孤岛（cross-silo）设置设计，我们同时展示了FedHBM在中高规模跨设备场景下的适用性，以及如何利用优质模型初始化（如预训练）实现快速加速。在大规模真实联邦数据集上的扩展实验进一步验证了该方法在现实FL应用中的有效性。