Stochastic bilevel optimization, which captures the inherent nested structure of machine learning problems, is gaining popularity in many recent applications. Existing works on bilevel optimization mostly consider either unconstrained problems or constrained upper-level problems. This paper considers the stochastic bilevel optimization problems with equality constraints both in the upper and lower levels. By leveraging the special structure of the equality constraints problem, the paper first presents an alternating implicit projected SGD approach and establishes the $\tilde{\cal O}(\epsilon^{-2})$ sample complexity that matches the state-of-the-art complexity of ALSET \citep{chen2021closing} for unconstrained bilevel problems. To further save the cost of projection, the paper presents two alternating implicit projection-efficient SGD approaches, where one algorithm enjoys the $\tilde{\cal O}(\epsilon^{-2}/T)$ upper-level and $\tilde{\cal O}(\epsilon^{-1.5}/T^{\frac{3}{4}})$ lower-level projection complexity with ${\cal O}(T)$ lower-level batch size, and the other one enjoys $\tilde{\cal O}(\epsilon^{-1.5})$ upper-level and lower-level projection complexity with ${\cal O}(1)$ batch size. Application to federated bilevel optimization has been presented to showcase the empirical performance of our algorithms. Our results demonstrate that equality-constrained bilevel optimization with strongly-convex lower-level problems can be solved as efficiently as stochastic single-level optimization problems.

翻译：随机双层优化捕捉了机器学习问题固有的嵌套结构，在近期众多应用中日益流行。现有双层优化研究大多考虑无约束问题或仅上层带约束的问题。本文考虑上下层均带有等式约束的随机双层优化问题。通过利用等式约束问题的特殊结构，本文首先提出了一种交替隐式投影随机梯度下降方法，并建立了$\tilde{\cal O}(\epsilon^{-2})$的样本复杂度，与无约束双层优化中最先进的ALSET方法\citep{chen2021closing}相匹配。为进一步降低投影计算成本，本文提出了两种交替隐式投影高效随机梯度下降方法。其中一种算法在$\mathcal{O}(T)$下层批量大小下，实现了$\tilde{\cal O}(\epsilon^{-2}/T)$的上层投影复杂度和$\tilde{\cal O}(\epsilon^{-1.5}/T^{\frac{3}{4}})$的下层投影复杂度；另一种算法在$\mathcal{O}(1)$批量大小下，实现了$\tilde{\cal O}(\epsilon^{-1.5})$的上、下层投影复杂度。我们通过联邦双层优化应用展示了所提算法的实证性能。结果表明：在强凸下层问题条件下，等式约束双层优化问题可像随机单层优化问题一样高效求解。