Bilevel optimization is one of the fundamental problems in machine learning and optimization. Recent theoretical developments in bilevel optimization focus on finding the first-order stationary points for nonconvex-strongly-convex cases. In this paper, we analyze algorithms that can escape saddle points in nonconvex-strongly-convex bilevel optimization. Specifically, we show that the perturbed approximate implicit differentiation (AID) with a warm start strategy finds $\epsilon$-approximate local minimum of bilevel optimization in $\tilde{O}(\epsilon^{-2})$ iterations with high probability. Moreover, we propose an inexact NEgative-curvature-Originated-from-Noise Algorithm (iNEON), a pure first-order algorithm that can escape saddle point and find local minimum of stochastic bilevel optimization. As a by-product, we provide the first nonasymptotic analysis of perturbed multi-step gradient descent ascent (GDmax) algorithm that converges to local minimax point for minimax problems.

翻译：双层优化是机器学习和优化领域中的基本问题之一。近期关于双层优化的理论进展主要聚焦于非凸-强凸情形下的一阶驻点寻找。本文分析了能够在非凸-强凸双层优化中逃离鞍点的算法。具体而言，我们证明了采用热启动策略的扰动近似隐式微分（AID）算法能够在 $\tilde{O}(\epsilon^{-2})$ 次迭代内以高概率找到双层优化的 $\epsilon$-近似局部最小值。此外，我们提出了一种不精确的噪声衍生负曲率算法（iNEON），这是一种纯一阶算法，能够逃离鞍点并找到随机双层优化的局部最小值。作为副产品，我们首次给出了扰动多步梯度下降上升（GDmax）算法的非渐近分析，该算法能够收敛到极小极大问题的局部极小极大点。