Recent Newton-type federated learning algorithms have demonstrated linear convergence with respect to the communication rounds. However, communicating Hessian matrices is often unfeasible due to their quadratic communication complexity. In this paper, we introduce a novel approach to tackle this issue while still achieving fast convergence rates. Our proposed method, named as Federated Newton Sketch methods (FedNS), approximates the centralized Newton's method by communicating the sketched square-root Hessian instead of the exact Hessian. To enhance communication efficiency, we reduce the sketch size to match the effective dimension of the Hessian matrix. We provide convergence analysis based on statistical learning for the federated Newton sketch approaches. Specifically, our approaches reach super-linear convergence rates w.r.t. the communication rounds for the first time. We validate the effectiveness of our algorithms through various experiments, which coincide with our theoretical findings.

翻译：近年来，基于牛顿型的联邦学习算法在通信轮次上展现出线性收敛性。然而，海森矩阵的通信因其二次通信复杂度而往往不可行。本文提出一种新方法以解决该问题，同时保持快速收敛速率。我们提出的方法名为联邦牛顿草图方法（FedNS），通过传输草图化平方根海森矩阵而非精确海森矩阵来近似集中式牛顿法。为提升通信效率，我们将草图尺寸缩减至海森矩阵的有效维度。基于统计学习理论，我们为联邦牛顿草图方法提供了收敛性分析。具体而言，我们的方法首次在通信轮次上达到超线性收敛速率。通过多项实验验证了算法的有效性，实验结果与理论分析一致。