Bilevel optimization reveals the inner structure of otherwise oblique optimization problems, such as hyperparameter tuning and meta-learning. A common goal in bilevel optimization is to find stationary points of the hyper-objective function. Although this hyper-objective approach is widely used, its theoretical properties have not been thoroughly investigated in cases where the lower-level functions lack strong convexity. In this work, we take a step forward and study the hyper-objective approach without the typical lower-level strong convexity assumption. Our hardness results show that the hyper-objective of general convex lower-level functions can be intractable either to evaluate or to optimize. To tackle this challenge, we introduce the gradient dominant condition, which strictly relaxes the strong convexity assumption by allowing the lower-level solution set to be non-singleton. Under the gradient dominant condition, we propose the Inexact Gradient-Free Method (IGFM), which uses the Switching Gradient Method (SGM) as the zeroth order oracle, to find an approximate stationary point of the hyper-objective. We also extend our results to nonsmooth lower-level functions under the weak sharp minimum condition.

翻译：双层优化揭示了超参数调优与元学习等复杂优化问题的内在结构。其核心目标通常在于寻找超目标函数的驻点。尽管超目标方法被广泛采用，但在下层函数缺乏强凸性时，其理论性质尚未得到充分探索。本文基于超目标视角，突破经典下层强凸性假设的局限，首次系统研究了该方法的理论边界。我们的困难性结果表明，对于一般凸下层函数，其超目标可能在求值或优化方面均难以处理。为解决这一挑战，我们引入梯度主导条件，该条件通过允许下层解集非单值来严格松弛强凸性假设。在此条件下，我们提出非精确无梯度方法（IGFM），该方法采用切换梯度方法（SGM）作为零阶预言机，以寻找超目标的近似驻点。此外，我们将研究结果推广至满足弱尖点最小值条件的非光滑下层函数场景。