Federated learning is a distributed learning framework that allows a set of clients to collaboratively train a model under the orchestration of a central server, without sharing raw data samples. Although in many practical scenarios the derivatives of the objective function are not available, only few works have considered the federated zeroth-order setting, in which functions can only be accessed through a budgeted number of point evaluations. In this work we focus on convex optimization and design the first federated zeroth-order algorithm to estimate the curvature of the global objective, with the purpose of achieving superlinear convergence. We take an incremental Hessian estimator whose error norm converges linearly, and we adapt it to the federated zeroth-order setting, sampling the random search directions from the Stiefel manifold for improved performance. In particular, both the gradient and Hessian estimators are built at the central server in a communication-efficient and privacy-preserving way by leveraging synchronized pseudo-random number generators. We provide a theoretical analysis of our algorithm, named FedZeN, proving local quadratic convergence with high probability and global linear convergence up to zeroth-order precision. Numerical simulations confirm the superlinear convergence rate and show that our algorithm outperforms the federated zeroth-order methods available in the literature.

翻译：联邦学习是一种分布式学习框架，允许一组客户端在中央服务器的协调下协作训练模型，而无需共享原始数据样本。尽管在许多实际场景中目标函数的导数不可用，但仅有少数研究考虑了联邦零阶设置——在此设置中，函数只能通过有限次数的点评估进行访问。本文聚焦于凸优化问题，设计了首个能够估计全局目标曲率的联邦零阶算法，旨在实现超线性收敛。我们采用了一种误差范数线性收敛的增量Hessian估计器，并将其适配至联邦零阶设置：通过从Stiefel流形中采样随机搜索方向以提升性能。特别地，梯度与Hessian估计器均在中央服务器以通信高效且保护隐私的方式构建，其实现依赖于同步伪随机数生成器。我们对所提出的算法FedZeN进行了理论分析，证明其以高概率达到局部二次收敛，并在零阶精度范围内实现全局线性收敛。数值模拟验证了其超线性收敛速率，并表明该算法优于现有文献中的联邦零阶方法。