This paper studies the complexity of finding an $ε$-stationary point for stochastic bilevel optimization when the upper-level problem is nonconvex and the lower-level problem is strongly convex. Recent work proposed the first-order method, F${}^2$SA, achieving the $\tilde{\mathcal{O}}(ε^{-6})$ upper complexity bound for first-order smooth problems. This is slower than the optimal $Ω(ε^{-4})$ complexity lower bound in its single-level counterpart. In this work, we show that faster rates are achievable for higher-order smooth problems. We first reformulate F$^2$SA as approximating the hyper-gradient with a forward difference. Based on this observation, we propose a class of methods F${}^2$SA-$p$ that uses $p$th-order finite difference for hyper-gradient approximation and improves the upper bound to $\tilde{\mathcal{O}}(p ε^{-4-p/2})$ for $p$th-order smooth problems. Finally, we demonstrate that the $Ω(ε^{-4})$ lower bound also holds for stochastic bilevel problems when the high-order smoothness holds for the lower-level variable, indicating that the upper bound of F${}^2$SA-$p$ is nearly optimal in the highly smooth region $p = Ω( \log ε^{-1} / \log \log ε^{-1})$.

翻译：本文研究当上层问题非凸且下层问题强凸时，随机双层优化中寻找$ε$-稳定点的复杂度。近期工作提出了F${}^2$SA一阶方法，在一阶光滑问题上达到了$\tilde{\mathcal{O}}(ε^{-6})$的复杂度上界。这比其单层问题对应的最优下界$Ω(ε^{-4})$更慢。本文证明，对于更高阶光滑问题，可以实现更快的收敛速率。我们首先将F$^2$SA重新表述为用前向差分近似超梯度。基于这一观察，我们提出了一类方法F${}^2$SA-$p$，该方法使用$p$阶有限差分进行超梯度近似，并将$p$阶光滑问题的上界改进为$\tilde{\mathcal{O}}(p ε^{-4-p/2})$。最后，我们证明当高阶光滑性对下层变量成立时，$Ω(ε^{-4})$下界同样适用于随机双层问题，这表明在高光滑区域$p = Ω( \log ε^{-1} / \log \log ε^{-1})$中，F${}^2$SA-$p$的上界几乎是紧的。