Bilevel Optimization has witnessed notable progress recently with new emerging efficient algorithms. However, its application in the Federated Learning setting remains relatively underexplored, and the impact of Federated Learning's inherent challenges on the convergence of bilevel algorithms remain obscure. In this work, we investigate Federated Bilevel Optimization problems and propose a communication-efficient algorithm, named FedBiOAcc. The algorithm leverages an efficient estimation of the hyper-gradient in the distributed setting and utilizes the momentum-based variance-reduction acceleration. Remarkably, FedBiOAcc achieves a communication complexity $O(\epsilon^{-1})$, a sample complexity $O(\epsilon^{-1.5})$ and the linear speed up with respect to the number of clients. We also analyze a special case of the Federated Bilevel Optimization problems, where lower level problems are locally managed by clients. We prove that FedBiOAcc-Local, a modified version of FedBiOAcc, converges at the same rate for this type of problems. Finally, we validate the proposed algorithms through two real-world tasks: Federated Data-cleaning and Federated Hyper-representation Learning. Empirical results show superior performance of our algorithms.

翻译：双层优化近期随着新涌现的高效算法取得了显著进展。然而，其在联邦学习环境中的应用仍相对未被充分探索，且联邦学习固有挑战对双层算法收敛性的影响尚不明确。本文研究了联邦双层优化问题，并提出了一种名为FedBiOAcc的通信高效算法。该算法利用分布式环境下对超梯度的有效估计，并采用基于动量的方差缩减加速技术。值得注意的是，FedBiOAcc实现了通信复杂度$O(\epsilon^{-1})$、样本复杂度$O(\epsilon^{-1.5})$以及随客户端数量的线性加速。我们还分析了联邦双层优化问题的一个特例，即下层问题由客户端本地管理。我们证明该类型问题下的改进版本FedBiOAcc-Local以相同收敛速率收敛。最后，通过联邦数据清洗与联邦超表征学习两项实际任务验证了所提算法，实验结果表明我们的算法具有优越性能。