Bilevel Optimization has witnessed notable progress recently with new emerging efficient algorithms, yet it is underexplored in the Federated Learning setting. It is unclear how the challenges of Federated Learning affect the convergence of bilevel algorithms. In this work, we study Federated Bilevel Optimization problems. We first propose the FedBiO algorithm that solves the hyper-gradient estimation problem efficiently, then we propose FedBiOAcc to accelerate FedBiO. FedBiO has communication complexity $O(\epsilon^{-1.5})$ with linear speed up, while FedBiOAcc achieves communication complexity $O(\epsilon^{-1})$, sample complexity $O(\epsilon^{-1.5})$ and also the linear speed up. We also study Federated Bilevel Optimization problems with local lower level problems, and prove that FedBiO and FedBiOAcc converges at the same rate with some modification.

翻译：双层优化近期随着新型高效算法的涌现取得了显著进展，但在联邦学习场景中仍未被充分探索。联邦学习的挑战如何影响双层算法的收敛性尚不明确。本文研究联邦双层优化问题。我们首先提出FedBiO算法，该算法能高效处理超梯度估计问题，随后提出加速版本FedBiOAcc。FedBiO的通信复杂度为$O(\epsilon^{-1.5})$且具备线性加速比，而FedBiOAcc实现了$O(\epsilon^{-1})$的通信复杂度、$O(\epsilon^{-1.5})$的样本复杂度以及线性加速比。我们还研究了带有局部下层问题的联邦双层优化问题，并证明通过适当修改，FedBiO与FedBiOAcc在该场景下仍保持相同的收敛速率。