Federated bilevel optimization (FBO) has shown great potential recently in machine learning and edge computing due to the emerging nested optimization structure in meta-learning, fine-tuning, hyperparameter tuning, etc. However, existing FBO algorithms often involve complicated computations and require multiple sub-loops per iteration, each of which contains a number of communication rounds. In this paper, we propose a simple and flexible FBO framework named SimFBO, which is easy to implement without sub-loops, and includes a generalized server-side aggregation and update for improving communication efficiency. We further propose System-level heterogeneity robust FBO (ShroFBO) as a variant of SimFBO with stronger resilience to heterogeneous local computation. We show that SimFBO and ShroFBO provably achieve a linear convergence speedup with partial client participation and client sampling without replacement, as well as improved sample and communication complexities. Experiments demonstrate the effectiveness of the proposed methods over existing FBO algorithms.

翻译：去中心化双层优化（FBO）近年来在机器学习与边缘计算领域展现出巨大潜力，这源于元学习、微调、超参数调优等过程中涌现的嵌套优化结构。然而，现有FBO算法通常涉及复杂计算，每次迭代需要多个子循环，每个子循环包含大量通信轮次。本文提出一种名为SimFBO的简单灵活的去中心化双层优化框架，其无需子循环即可轻松实现，并通过通用化的服务器端聚合与更新机制提升通信效率。我们进一步提出系统级异构鲁棒去中心化双层优化（ShroFBO）作为SimFBO的变体，其能更强地适应异构本地计算。理论证明表明，SimFBO与ShroFBO在部分客户端参与和无放回客户端采样场景下可达到线性收敛加速比，同时具有更优的样本复杂度与通信复杂度。实验验证了所提方法相较现有FBO算法的有效性。