Federated Averaging (FedAvg) is known to experience convergence issues when encountering significant clients system heterogeneity and data heterogeneity. Server momentum has been proposed as an effective mitigation. However, existing server momentum works are restrictive in the momentum formulation, do not properly schedule hyperparameters and focus only on system homogeneous settings, which leaves the role of server momentum still an under-explored problem. In this paper, we propose a general framework for server momentum, that (a) covers a large class of momentum schemes that are unexplored in federated learning (FL), (b) enables a popular stagewise hyperparameter scheduler, (c) allows heterogeneous and asynchronous local computing. We provide rigorous convergence analysis for the proposed framework. To our best knowledge, this is the first work that thoroughly analyzes the performances of server momentum with a hyperparameter scheduler and system heterogeneity. Extensive experiments validate the effectiveness of our proposed framework.

翻译：联邦平均（FedAvg）在面临显著的系统异构性和数据异构性时，已知会出现收敛问题。服务器动量已被提出作为一种有效的缓解措施。然而，现有的服务器动量工作在动量公式上具有局限性，未能合理调度超参数，且仅关注系统同构设置，这使得服务器动量的作用仍是一个尚未充分探索的问题。本文提出了一种通用的服务器动量框架，该框架：（a）涵盖了联邦学习（FL）中未被探索的大类动量方案，（b）实现了流行的阶段式超参数调度器，（c）允许异构和异步的本地计算。我们对所提出的框架提供了严格的收敛性分析。据我们所知，这是首个全面分析带超参数调度器和系统异构性的服务器动量性能的工作。大量实验验证了我们提出框架的有效性。