Bilevel optimization has recently attracted considerable attention due to its abundant applications in machine learning problems. However, existing methods rely on prior knowledge of problem parameters to determine stepsizes, resulting in significant effort in tuning stepsizes when these parameters are unknown. In this paper, we propose two novel tuning-free algorithms, D-TFBO and S-TFBO. D-TFBO employs a double-loop structure with stepsizes adaptively adjusted by the "inverse of cumulative gradient norms" strategy. S-TFBO features a simpler fully single-loop structure that updates three variables simultaneously with a theory-motivated joint design of adaptive stepsizes for all variables. We provide a comprehensive convergence analysis for both algorithms and show that D-TFBO and S-TFBO respectively require $O(\frac{1}{\epsilon})$ and $O(\frac{1}{\epsilon}\log^4(\frac{1}{\epsilon}))$ iterations to find an $\epsilon$-accurate stationary point, (nearly) matching their well-tuned counterparts using the information of problem parameters. Experiments on various problems show that our methods achieve performance comparable to existing well-tuned approaches, while being more robust to the selection of initial stepsizes. To the best of our knowledge, our methods are the first to completely eliminate the need for stepsize tuning, while achieving theoretical guarantees.

翻译：双层优化因其在机器学习问题中的广泛应用，近期受到广泛关注。然而，现有方法依赖问题参数的先验知识来确定步长，当这些参数未知时，需要耗费大量精力进行步长调参。本文提出了两种新颖的免调参算法：D-TFBO 与 S-TFBO。D-TFBO 采用双循环结构，其步长通过“累积梯度范数倒数”策略自适应调整。S-TFBO 则采用更简洁的完全单循环结构，同时更新三个变量，并为所有变量设计了理论驱动的自适应步长联合方案。我们对两种算法进行了全面的收敛性分析，证明 D-TFBO 和 S-TFBO 分别需要 $O(\frac{1}{\epsilon})$ 和 $O(\frac{1}{\epsilon}\log^4(\frac{1}{\epsilon}))$ 次迭代以找到一个 $\epsilon$-精度的稳定点，其性能（近乎）匹配了利用问题参数信息进行充分调参的对应算法。在多种问题上的实验表明，我们的方法在达到与现有充分调参方法相当性能的同时，对初始步长的选择具有更强的鲁棒性。据我们所知，我们的方法是首个在完全无需步长调参的同时，仍能保证理论性能的方法。