Federated Learning (FL) is a distributed machine learning approach that enables model training in communication efficient and privacy-preserving manner. The standard optimization method in FL is Federated Averaging (FedAvg), which performs multiple local SGD steps between communication rounds. FedAvg has been considered to lack algorithm adaptivity compared to modern first-order adaptive optimizations. In this paper, we propose new communication-efficient FL algortithms based on two adaptive frameworks: local adaptivity (PreFed) and server-side adaptivity (PreFedOp). Proposed methods adopt adaptivity by using a novel covariance matrix preconditioner. Theoretically, we provide convergence guarantees for our algorithms. The empirical experiments show our methods achieve state-of-the-art performances on both i.i.d. and non-i.i.d. settings.

翻译：联邦学习（FL）是一种分布式机器学习方法，能够以通信高效且保护隐私的方式实现模型训练。FL中的标准优化方法是联邦平均算法（FedAvg），其在通信轮次之间执行多次本地SGD步骤。与现代化的自适应一阶优化算法相比，FedAvg被认为缺乏算法自适应性。本文基于两种自适应框架提出新的通信高效联邦学习算法：本地自适应性（PreFed）与服务器端自适应性（PreFedOp）。所提方法通过使用新型协方差矩阵预条件器实现自适应性。理论方面，我们为算法提供了收敛性保证。实证实验表明，我们的方法在独立同分布和非独立同分布场景下均取得了最先进的性能表现。