This paper presents a new approach and algorithm for solving a class of constrained Bi-Level Optimization (BLO) problems in which the lower-level problem involves constraints coupling both upper-level and lower-level variables. Such problems have recently gained significant attention due to their broad applicability in machine learning. However, conventional gradient-based methods unavoidably rely on computationally intensive calculations related to the Hessian matrix. To address this challenge, we begin by devising a smooth proximal Lagrangian value function to handle the constrained lower-level problem. Utilizing this construct, we introduce a single-level reformulation for constrained BLOs that transforms the original BLO problem into an equivalent optimization problem with smooth constraints. Enabled by this reformulation, we develop a Hessian-free gradient-based algorithm-termed proximal Lagrangian Value function-based Hessian-free Bi-level Algorithm (LV-HBA)-that is straightforward to implement in a single loop manner. Consequently, LV-HBA is especially well-suited for machine learning applications. Furthermore, we offer non-asymptotic convergence analysis for LV-HBA, eliminating the need for traditional strong convexity assumptions for the lower-level problem while also being capable of accommodating non-singleton scenarios. Empirical results substantiate the algorithm's superior practical performance.

翻译：本文提出了一种新的方法和算法，用于求解一类约束双层优化问题，其中下层问题涉及耦合上层和下层变量的约束。这类问题因其在机器学习中的广泛适用性而近年来备受关注。然而，传统的基于梯度的方法不可避免地依赖于与海森矩阵相关的计算密集型操作。为解决这一挑战，我们首先设计了一种光滑的近端拉格朗日价值函数来处理约束下层问题。利用这一构造，我们引入了一种针对约束双层优化的单层重构，将原始双层优化问题转化为一个具有光滑约束的等价优化问题。基于这一重构，我们开发了一种免海森矩阵的梯度算法——称为基于近端拉格朗日价值函数的免海森双层算法——该算法易于以单循环方式实现。因此，LV-HBA特别适用于机器学习应用。此外，我们提供了LV-HBA的非渐近收敛性分析，该分析无需对下层问题进行传统的强凸性假设，同时也能够处理非单点集场景。实验结果验证了该算法卓越的实际性能。