Federated Averaging (FedAvg) remains the most popular algorithm for Federated Learning (FL) optimization due to its simple implementation, stateless nature, and privacy guarantees combined with secure aggregation. Recent work has sought to generalize the vanilla averaging in FedAvg to a generalized gradient descent step by treating client updates as pseudo-gradients and using a server step size. While the use of a server step size has been shown to provide performance improvement theoretically, the practical benefit of the server step size has not been seen in most existing works. In this work, we present FedExP, a method to adaptively determine the server step size in FL based on dynamically varying pseudo-gradients throughout the FL process. We begin by considering the overparameterized convex regime, where we reveal an interesting similarity between FedAvg and the Projection Onto Convex Sets (POCS) algorithm. We then show how FedExP can be motivated as a novel extension to the extrapolation mechanism that is used to speed up POCS. Our theoretical analysis later also discusses the implications of FedExP in underparameterized and non-convex settings. Experimental results show that FedExP consistently converges faster than FedAvg and competing baselines on a range of realistic FL datasets.

翻译：联邦平均（FedAvg）因其实现简单、无状态特性及结合安全聚合的隐私保障，仍是联邦学习（FL）优化中最流行的算法。近期研究尝试将FedAvg中的朴素平均推广为广义梯度下降步骤，通过将客户端更新视为伪梯度并采用服务器步长。尽管理论上表明服务器步长能带来性能提升，但现有多数工作并未观察到其实际益处。本文提出FedExP——一种基于联邦学习过程中动态变化的伪梯度自适应确定服务器步长的方法。我们首先考虑过参数化凸场景，在此情境下揭示FedAvg与凸集投影（POCS）算法之间存在有趣的相似性，进而论证FedExP如何作为加速POCS的外推机制的新型扩展被提出。随后的理论分析进一步讨论了FedExP在欠参数化及非凸场景下的影响。实验结果表明，在多种真实联邦学习数据集上，FedExP的收敛速度始终优于FedAvg及竞争性基线方法。