Bilevel optimization is a popular two-level hierarchical optimization, which has been widely applied to many machine learning tasks such as hyperparameter learning, meta learning and continual learning. Although many bilevel optimization methods recently have been developed, the bilevel methods are not well studied when the lower-level problem is nonconvex. To fill this gap, in the paper, we study a class of nonconvex bilevel optimization problems, which both upper-level and lower-level problems are nonconvex, and the lower-level problem satisfies Polyak-Lojasiewicz (PL) condition. We propose an efficient momentum-based gradient bilevel method (MGBiO) to solve these deterministic problems. Meanwhile, we propose a class of efficient momentum-based stochastic gradient bilevel methods (MSGBiO and VR-MSGBiO) to solve these stochastic problems. Moreover, we provide a useful convergence analysis framework for our methods. Specifically, under some mild conditions, we prove that our MGBiO method has a sample (or gradient) complexity of $O(\epsilon^{-2})$ for finding an $\epsilon$-stationary solution of the deterministic bilevel problems (i.e., $\|\nabla F(x)\|\leq \epsilon$), which improves the existing best results by a factor of $O(\epsilon^{-1})$. Meanwhile, we prove that our MSGBiO and VR-MSGBiO methods have sample complexities of $\tilde{O}(\epsilon^{-4})$ and $\tilde{O}(\epsilon^{-3})$, respectively, in finding an $\epsilon$-stationary solution of the stochastic bilevel problems (i.e., $\mathbb{E}\|\nabla F(x)\|\leq \epsilon$), which improves the existing best results by a factor of $O(\epsilon^{-3})$. This manuscript commemorates the mathematician Boris Polyak (1935 -2023).

翻译：双层优化是一种流行的双层分层优化方法，已被广泛应用于超参数学习、元学习和持续学习等机器学习任务。尽管近期已发展出许多双层优化方法，但当下层问题为非凸时，这些方法的理论研究仍不充分。为填补这一空白，本文研究了一类非凸双层优化问题——其上下层问题均为非凸，且下层问题满足 Polyak-Lojasiewicz (PL) 条件。针对确定性情形，我们提出了一种高效的动量梯度双层优化方法（MGBiO）；针对随机情形，我们进一步提出了动量随机梯度双层优化方法（MSGBiO 及 VR-MSGBiO）。此外，我们为这些方法建立了一套有效的收敛性分析框架。具体而言，在温和条件下，我们证明 MGBiO 方法在寻找确定性双层问题的 $\epsilon$-稳定解（即 $\|\nabla F(x)\|\leq \epsilon$）时，采样（或梯度）复杂度为 $O(\epsilon^{-2})$，较现有最优结果改进了一个 $O(\epsilon^{-1})$ 因子。同时，我们证明 MSGBiO 和 VR-MSGBiO 方法在寻找随机双层问题的 $\epsilon$-稳定解（即 $\mathbb{E}\|\nabla F(x)\|\leq \epsilon$）时，采样复杂度分别为 $\tilde{O}(\epsilon^{-4})$ 和 $\tilde{O}(\epsilon^{-3})$，较现有最优结果改进了一个 $O(\epsilon^{-3})$ 因子。本文谨纪念数学家 Boris Polyak（1935–2023）。