Federated Learning (FL) is a nascent decentralized learning framework under which a massive collection of heterogeneous clients collaboratively train a model without revealing their local data. Scarce communication, privacy leakage, and Byzantine attacks are the key bottlenecks of system scalability. In this paper, we focus on communication-efficient distributed (stochastic) gradient descent for non-convex optimization, a driving force of FL. We propose two algorithms, named {\em Adaptive Stochastic Sign SGD (Ada-StoSign)} and {\em $\beta$-Stochastic Sign SGD ($\beta$-StoSign)}, each of which compresses the local gradients into bit vectors. To handle unbounded gradients, Ada-StoSign uses a novel norm tracking function that adaptively adjusts a coarse estimation on the $\ell_{\infty}$ of the local gradients - a key parameter used in gradient compression. We show that Ada-StoSign converges in expectation with a rate $O(\log T/\sqrt{T} + 1/\sqrt{M})$, where $M$ is the number of clients. To the best of our knowledge, when $M$ is sufficiently large, Ada-StoSign outperforms the state-of-the-art sign-based method whose convergence rate is $O(T^{-1/4})$. Under bounded gradient assumption, $\beta$-StoSign achieves quantifiable Byzantine resilience and privacy assurances, and works with partial client participation and mini-batch gradients which could be unbounded. We corroborate and complement our theories by experiments on MNIST and CIFAR-10 datasets.

翻译：联邦学习（FL）是一种新兴的去中心化学习框架，其中大量异构客户端在不泄露本地数据的情况下协同训练模型。通信资源稀缺、隐私泄露和拜占庭攻击是系统可扩展性的关键瓶颈。本文关注通信高效的分布式（随机）梯度下降法在非凸优化问题中的应用——这是联邦学习的核心驱动力。我们提出两种算法，分别命名为"自适应随机符号SGD（Ada-StoSign）"和"β-随机符号SGD（β-StoSign）"，每种算法都将局部梯度压缩为比特向量。为处理无界梯度，Ada-StoSign采用新颖的范数追踪函数，自适应调整局部梯度ℓ∞范数的粗估计——这是梯度压缩中的关键参数。我们证明Ada-StoSign以速率O(log T/√T + 1/√M)依期望收敛，其中M为客户端数量。据我们所知，当M足够大时，Ada-StoSign的性能优于收敛率为O(T^{-1/4})的现有最优符号方法。在有界梯度假设下，β-StoSign可实现可量化的拜占庭鲁棒性与隐私保障，并支持部分客户端参与和可能无界的迷你批次梯度。我们通过在MNIST和CIFAR-10数据集上的实验验证并补充了理论分析。