In this paper, we study a class of stochastic bilevel optimization problems, also known as stochastic simple bilevel optimization, where we minimize a smooth stochastic objective function over the optimal solution set of another stochastic convex optimization problem. We introduce novel stochastic bilevel optimization methods that locally approximate the solution set of the lower-level problem via a stochastic cutting plane, and then run a conditional gradient update with variance reduction techniques to control the error induced by using stochastic gradients. For the case that the upper-level function is convex, our method requires $\tilde{\mathcal{O}}(\max\{1/\epsilon_f^{2},1/\epsilon_g^{2}\}) $ stochastic oracle queries to obtain a solution that is $\epsilon_f$-optimal for the upper-level and $\epsilon_g$-optimal for the lower-level. This guarantee improves the previous best-known complexity of $\mathcal{O}(\max\{1/\epsilon_f^{4},1/\epsilon_g^{4}\})$. Moreover, for the case that the upper-level function is non-convex, our method requires at most $\tilde{\mathcal{O}}(\max\{1/\epsilon_f^{3},1/\epsilon_g^{3}\}) $ stochastic oracle queries to find an $(\epsilon_f, \epsilon_g)$-stationary point. In the finite-sum setting, we show that the number of stochastic oracle calls required by our method are $\tilde{\mathcal{O}}(\sqrt{n}/\epsilon)$ and $\tilde{\mathcal{O}}(\sqrt{n}/\epsilon^{2})$ for the convex and non-convex settings, respectively, where $\epsilon=\min \{\epsilon_f,\epsilon_g\}$.

翻译：本文研究一类随机双层优化问题，也称为随机简单双层优化，其目标是在另一个随机凸优化问题的最优解集上最小化一个光滑随机目标函数。我们提出了一种新颖的随机双层优化方法，该方法通过随机切割平面局部逼近下层问题的解集，然后采用结合方差缩减技术的条件梯度更新来控制使用随机梯度所引入的误差。对于上层函数为凸的情形，我们的方法需要 $\tilde{\mathcal{O}}(\max\{1/\epsilon_f^{2},1/\epsilon_g^{2}\}) $ 次随机 oracle 查询，即可得到上层 $\epsilon_f$-最优且下层 $\epsilon_g$-最优的解。这一结果改进了先前已知的最优复杂度 $\mathcal{O}(\max\{1/\epsilon_f^{4},1/\epsilon_g^{4}\})$。此外，对于上层函数非凸的情形，我们的方法最多需要 $\tilde{\mathcal{O}}(\max\{1/\epsilon_f^{3},1/\epsilon_g^{3}\}) $ 次随机 oracle 查询即可找到 $(\epsilon_f, \epsilon_g)$-稳定点。在有限和设定下，我们证明该方法所需的随机 oracle 调用次数在凸和非凸情形下分别为 $\tilde{\mathcal{O}}(\sqrt{n}/\epsilon)$ 和 $\tilde{\mathcal{O}}(\sqrt{n}/\epsilon^{2})$，其中 $\epsilon=\min \{\epsilon_f,\epsilon_g\}$。