Cross-device training is a crucial subfield of federated learning, where the number of clients can reach into the billions. Standard approaches and local methods are prone to issues such as client drift and insensitivity to data similarities. We propose a novel algorithm (SPAM) for cross-device federated learning with non-convex losses, which solves both issues. We provide sharp analysis under second-order (Hessian) similarity, a condition satisfied by a variety of machine learning problems in practice. Additionally, we extend our results to the partial participation setting, where a cohort of selected clients communicate with the server at each communication round. Our method is the first in its kind, that does not require the smoothness of the objective and provably benefits from clients having similar data.

翻译：跨设备训练是联邦学习的关键子领域，其客户端数量可达数十亿规模。标准方法与局部方法易受客户端漂移及对数据相似性不敏感等问题影响。本文针对非凸损失函数的跨设备联邦学习提出一种新颖算法（SPAM），可同时解决上述两类问题。我们在二阶（海森）相似性条件下给出精确分析，该条件在实践中被多种机器学习问题所满足。此外，我们将结果扩展至部分参与场景，其中每轮通信仅由选定客户端群组与服务器进行交互。本方法作为该领域首创，既不要求目标函数的光滑性，又能从客户端数据相似性中获取可证明的收益。