In Federated Learning (FL), a number of clients or devices collaborate to train a model without sharing their data. Models are optimized locally at each client and further communicated to a central hub for aggregation. While FL is an appealing decentralized training paradigm, heterogeneity among data from different clients can cause the local optimization to drift away from the global objective. In order to estimate and therefore remove this drift, variance reduction techniques have been incorporated into FL optimization recently. However, these approaches inaccurately estimate the clients' drift and ultimately fail to remove it properly. In this work, we propose an adaptive algorithm that accurately estimates drift across clients. In comparison to previous works, our approach necessitates less storage and communication bandwidth, as well as lower compute costs. Additionally, our proposed methodology induces stability by constraining the norm of estimates for client drift, making it more practical for large scale FL. Experimental findings demonstrate that the proposed algorithm converges significantly faster and achieves higher accuracy than the baselines across various FL benchmarks.

翻译：在联邦学习中，多个客户端或设备协作训练模型，而无需共享数据。模型在每个客户端本地优化，随后被传输至中央枢纽进行聚合。尽管联邦学习是一种引人注目的分散式训练范式，但不同客户端数据间的异构性可能导致本地优化偏离全局目标。为估计并消除这种漂移，近期研究已将方差缩减技术融入联邦学习优化中。然而，这些方法对客户端漂移的估计不够精确，最终无法有效消除这一偏差。本研究提出一种自适应算法，能够准确估计各客户端的漂移。与先前工作相比，本方法所需存储空间、通信带宽及计算成本均更低。此外，所提方法通过约束客户端漂移估计的范数以增强稳定性，使其更适用于大规模联邦学习场景。实验结果表明，在多种联邦学习基准测试中，所提算法收敛速度显著更快，且精度高于基线方法。