We present a method for solving general nonconvex-strongly-convex bilevel optimization problems. Our method -- the \emph{Restarted Accelerated HyperGradient Descent} (\texttt{RAHGD}) method -- finds an $\epsilon$-first-order stationary point of the objective with $\tilde{\mathcal{O}}(\kappa^{3.25}\epsilon^{-1.75})$ oracle complexity, where $\kappa$ is the condition number of the lower-level objective and $\epsilon$ is the desired accuracy. We also propose a perturbed variant of \texttt{RAHGD} for finding an $\big(\epsilon,\mathcal{O}(\kappa^{2.5}\sqrt{\epsilon}\,)\big)$-second-order stationary point within the same order of oracle complexity. Our results achieve the best-known theoretical guarantees for finding stationary points in bilevel optimization and also improve upon the existing upper complexity bound for finding second-order stationary points in nonconvex-strongly-concave minimax optimization problems, setting a new state-of-the-art benchmark. Empirical studies are conducted to validate the theoretical results in this paper.

翻译：我们提出了一种求解一般非凸-强凸双层优化问题的方法。该方法——重启加速超梯度下降法（RAHGD）——以$\tilde{\mathcal{O}}(\kappa^{3.25}\epsilon^{-1.75})$的预言复杂度找到目标的$\epsilon$-一阶稳定点，其中$\kappa$是下层目标的条件数，$\epsilon$是期望精度。我们还提出了RAHGD的扰动变体，用于在相同的预言复杂度阶数内找到$\big(\epsilon,\mathcal{O}(\kappa^{2.5}\sqrt{\epsilon}\,)\big)$-二阶稳定点。我们的结果实现了双层优化中寻找稳定点的最佳已知理论保证，并改进了非凸-强凹极小极大优化问题中寻找二阶稳定点的现有上界复杂度，树立了新的最新基准。本文通过实证研究验证了理论结果。