In this paper, we consider the minimization of a nonsmooth nonconvex objective function $f(x)$ over a closed convex subset $\mathcal{X}$ of $\mathbb{R}^n$, with additional nonsmooth nonconvex constraints $c(x) = 0$. We develop a unified framework for developing Lagrangian-based methods, which takes a single-step update to the primal variables by some subgradient methods in each iteration. These subgradient methods are ``embedded'' into our framework, in the sense that they are incorporated as black-box updates to the primal variables. We prove that our proposed framework inherits the global convergence guarantees from these embedded subgradient methods under mild conditions. In addition, we show that our framework can be extended to solve constrained optimization problems with expectation constraints. Based on the proposed framework, we show that a wide range of existing stochastic subgradient methods, including the proximal SGD, proximal momentum SGD, and proximal ADAM, can be embedded into Lagrangian-based methods. Preliminary numerical experiments on deep learning tasks illustrate that our proposed framework yields efficient variants of Lagrangian-based methods with convergence guarantees for nonconvex nonsmooth constrained optimization problems.

翻译：本文考虑在$\mathbb{R}^n$的闭凸子集$\mathcal{X}$上最小化非光滑非凸目标函数$f(x)$，并带有额外非光滑非凸约束$c(x)=0$的问题。我们提出了一个统一的拉格朗日方法框架，该框架在每次迭代中通过某种次梯度方法对原始变量进行单步更新。这些次梯度方法被"嵌入"到我们的框架中，即它们作为原始变量的黑箱更新被整合进来。我们证明了在温和条件下，所提框架能够继承这些嵌入次梯度方法的全局收敛保证。此外，我们还表明该框架可扩展至求解带有期望约束的约束优化问题。基于该框架，我们证明了包括近端SGD、近端动量SGD和近端ADAM在内的多种现有随机次梯度方法均可嵌入到拉格朗日方法中。深度学习任务的初步数值实验表明，我们的框架能为非凸非光滑约束优化问题提供具有收敛保证的高效拉格朗日方法变体。