Federated learning has attracted increasing attention with the emergence of distributed data. While extensive federated learning algorithms have been proposed for the non-convex distributed problem, federated learning in practice still faces numerous challenges, such as the large training iterations to converge since the sizes of models and datasets keep increasing, and the lack of adaptivity by SGD-based model updates. Meanwhile, the study of adaptive methods in federated learning is scarce and existing works either lack a complete theoretical convergence guarantee or have slow sample complexity. In this paper, we propose an efficient adaptive algorithm (i.e., FAFED) based on the momentum-based variance-reduced technique in cross-silo FL. We first explore how to design the adaptive algorithm in the FL setting. By providing a counter-example, we prove that a simple combination of FL and adaptive methods could lead to divergence. More importantly, we provide a convergence analysis for our method and prove that our algorithm is the first adaptive FL algorithm to reach the best-known samples $O(\epsilon^{-3})$ and $O(\epsilon^{-2})$ communication rounds to find an $\epsilon$-stationary point without large batches. The experimental results on the language modeling task and image classification task with heterogeneous data demonstrate the efficiency of our algorithms.

翻译：联邦学习随着分布式数据的出现而日益受到关注。尽管针对非凸分布式问题已提出大量联邦学习算法，但实际应用中的联邦学习仍面临诸多挑战，例如由于模型和数据集的规模不断增大，训练迭代次数巨大才能收敛，以及基于SGD的模型更新缺乏自适应性。与此同时，联邦学习中自适应方法的研究较为稀缺，现有工作要么缺乏完整的理论收敛保证，要么样本复杂度较高。在本文中，我们提出一种基于动量方差缩减技术的高效自适应算法（即FAFED），适用于跨孤岛联邦学习场景。我们首先探索如何在联邦学习场景中设计自适应算法。通过提供一个反例，我们证明简单组合联邦学习与自适应方法可能导致发散。更重要的是，我们为所提方法提供收敛性分析，并证明我们的算法是首个无需大批量数据即可达到最优已知样本复杂度$O(\epsilon^{-3})$和通信轮次$O(\epsilon^{-2})$以找到$\epsilon$-稳定点的自适应联邦学习算法。在异构数据上的语言建模任务和图像分类任务的实验结果表明了我们算法的有效性。