We present in this paper novel accelerated fully first-order methods in \emph{Bilevel Optimization} (BLO). Firstly, for BLO under the assumption that the lower-level functions admit the typical strong convexity assumption, the \emph{(Perturbed) Restarted Accelerated Fully First-order methods for Bilevel Approximation} (\texttt{PRAF${}^2$BA}) algorithm leveraging \emph{fully} first-order oracles is proposed, whereas the algorithm for finding approximate first-order and second-order stationary points with state-of-the-art oracle query complexities in solving complex optimization tasks. Secondly, applying as a special case of BLO the \emph{nonconvex-strongly-convex} (NCSC) minimax optimization, \texttt{PRAF${}^2$BA} rediscovers \emph{perturbed restarted accelerated gradient descent ascent} (\texttt{PRAGDA}) that achieves the state-of-the-art complexity for finding approximate second-order stationary points. Additionally, we investigate the challenge of finding stationary points of the hyper-objective function in BLO when lower-level functions lack the typical strong convexity assumption, where we identify several regularity conditions of the lower-level problems that ensure tractability and present hardness results indicating the intractability of BLO for general convex lower-level functions. Under these regularity conditions we propose the \emph{Inexact Gradient-Free Method} (\texttt{IGFM}), utilizing the \emph{Switching Gradient Method} (\texttt{SGM}) as an efficient sub-routine to find an approximate stationary point of the hyper-objective in polynomial time. Empirical studies for real-world problems are provided to further validate the outperformance of our proposed algorithms.

翻译：本文提出了双层优化中新颖的加速全一阶方法。首先，针对下层函数满足典型强凸性假设的双层优化问题，我们提出了基于全一阶预言机的（扰动）重启加速全一阶双层逼近算法，该算法在求解复杂优化任务时能以最优的预言机查询复杂度找到近似一阶与二阶稳定点。其次，将非凸-强凸极小极大优化作为双层优化的特例，该算法重新发现了达到近似二阶稳定点最优复杂度的扰动重启加速梯度下降上升法。此外，我们研究了当下层函数缺乏典型强凸性假设时，双层优化中超目标函数稳定点的求解难题，识别了下层问题中确保可解性的若干正则性条件，并给出了关于一般凸下层函数双层优化问题难解性的硬度结果。在这些正则性条件下，我们提出了不精确无梯度方法，该方法以切换梯度法作为高效子程序，在多项式时间内找到超目标的近似稳定点。最后通过实际问题的实证研究进一步验证了所提算法的优越性能。