Bilevel optimization is a popular two-level hierarchical optimization, which has been widely applied to many machine learning tasks such as hyperparameter learning, meta learning and continual learning. Although many bilevel optimization methods recently have been developed, the bilevel methods are not well studied when the lower-level problem is nonconvex. To fill this gap, in the paper, we study a class of nonconvex bilevel optimization problems, where both upper-level and lower-level problems are nonconvex, and the lower-level problem satisfies Polyak-{\L}ojasiewicz (PL) condition. We propose an efficient momentum-based gradient bilevel method (MGBiO) to solve these deterministic problems. Meanwhile, we propose a class of efficient momentum-based stochastic gradient bilevel methods (MSGBiO and VR-MSGBiO) to solve these stochastic problems. Moreover, we provide a useful convergence analysis framework for our methods. Specifically, under some mild conditions, we prove that our MGBiO method has a sample (or gradient) complexity of $O(\epsilon^{-2})$ for finding an $\epsilon$-stationary solution of the deterministic bilevel problems (i.e., $\|\nabla F(x)\|\leq \epsilon$), which improves the existing best results by a factor of $O(\epsilon^{-1})$. Meanwhile, we prove that our MSGBiO and VR-MSGBiO methods have sample complexities of $\tilde{O}(\epsilon^{-4})$ and $\tilde{O}(\epsilon^{-3})$, respectively, in finding an $\epsilon$-stationary solution of the stochastic bilevel problems (i.e., $\mathbb{E}\|\nabla F(x)\|\leq \epsilon$), which improves the existing best results by a factor of $\tilde{O}(\epsilon^{-3})$. Extensive experimental results on bilevel PL game and hyper-representation learning demonstrate the efficiency of our algorithms. This paper commemorates the mathematician Boris Polyak (1935 -2023).

翻译：双层优化是一种流行的两层层次化优化方法，已广泛应用于超参数学习、元学习和持续学习等机器学习任务。尽管近年来已发展出许多双层优化方法，但当下层问题为非凸时，相关研究尚不充分。为填补这一空白，本文研究了一类非凸双层优化问题，其中上层和下层问题均为非凸，且下层问题满足Polyak-{\L}ojasiewicz (PL)条件。我们提出了一种高效的基于动量的梯度双层方法（MGBiO）来解决这些确定性优化问题。同时，我们提出了一类高效的基于动量的随机梯度双层方法（MSGBiO和VR-MSGBiO）来解决这些随机优化问题。此外，我们为所提方法提供了实用的收敛性分析框架。具体而言，在温和条件下，我们证明MGBiO方法在求解确定性双层问题的$\epsilon$-稳定解（即$\|\nabla F(x)\|\leq \epsilon$）时，其样本（或梯度）复杂度为$O(\epsilon^{-2})$，较现有最优结果提升了$O(\epsilon^{-1})$倍。同时，我们证明MSGBiO和VR-MSGBiO方法在求解随机双层问题的$\epsilon$-稳定解（即$\mathbb{E}\|\nabla F(x)\|\leq \epsilon$）时，其样本复杂度分别为$\tilde{O}(\epsilon^{-4})$和$\tilde{O}(\epsilon^{-3})$，较现有最优结果提升了$\tilde{O}(\epsilon^{-3})$倍。在双层PL博弈和超表征学习上的大量实验结果验证了我们算法的高效性。本文献给数学家Boris Polyak（1935—2023）以作纪念。