Bilevel optimization has been widely applied many machine learning problems such as hyperparameter optimization, policy optimization and meta learning. Although many bilevel optimization methods more recently have been proposed to solve the bilevel optimization problems, they still suffer from high computational complexities and do not consider the more general bilevel problems with nonsmooth regularization. In the paper, thus, we propose a class of efficient bilevel optimization methods based on Bregman distance. In our methods, we use the mirror decent iteration to solve the outer subproblem of the bilevel problem by using strongly-convex Bregman functions. Specifically, we propose a bilevel optimization method based on Bregman distance (BiO-BreD) for solving deterministic bilevel problems, which reaches the lower computational complexities than the best known results. We also propose a stochastic bilevel optimization method (SBiO-BreD) for solving stochastic bilevel problems based on the stochastic approximated gradients and Bregman distance. Further, we propose an accelerated version of SBiO-BreD method (ASBiO-BreD) by using the variance-reduced technique. Moreover, we prove that the ASBiO-BreD outperforms the best known computational complexities with respect to the condition number $\kappa$ and the target accuracy $\epsilon$ for finding an $\epsilon$-stationary point of nonconvex-strongly-convex bilevel problems. In particular, our methods can solve the bilevel optimization problems with nonsmooth regularization with a lower computational complexity.

翻译：双层优化已被广泛应用到许多机器学习问题,例如超参数优化、政策优化和元学习。尽管最近提出了许多双级优化方法以解决双级优化问题,但是它们仍然受到高计算复杂性的困扰,没有考虑到非单体正规化的更一般的双级问题。因此,在论文中,我们提议了一类基于Bregman距离的高效双级优化方法。在我们的方法中,我们使用镜像体面的循环来解决双级问题的外部子问题,方法是使用强对等的Bregman函数解决双级问题。具体地说,我们提议一种基于Bregman距离的双级优化方法,用于解决确定性双级优化问题,但基于Bregman距离(BiO-BreD) 的双级优化方法,这比已知的计算复杂性要低。我们还提议了Stoch 双级双级优化方法(SBIO-Breax) 的快速版本,用已知的ABI-BS-Rest 方法, 以我们已知的A-BS-Bredeal Ral Ral-Ser-Ser-Sergy 方法,用已知的Sal-Serviol-Sergy-Serviol-Serg-Serview ex ex ex ex 方法,用我们已知的Servical-rg-Serg-Serviol-de ex ex ex ex lax ex ex 一种已知的Serg-Servial-sal-de ex ex ex ex ex ex ex ex lax ex ex lax lax 的方法,用已知的Servicol ex ex ex lax lax a-col-col-col-col-cal ex ex a-chal ex ex ex ladgal) ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex