Selecting the best hyperparameters for a particular optimization instance, such as the learning rate and momentum, is an important but nonconvex problem. As a result, iterative optimization methods such as hypergradient descent lack global optimality guarantees in general. We propose an online nonstochastic control methodology for mathematical optimization. First, we formalize the setting of meta-optimization, an online learning formulation of learning the best optimization algorithm from a class of methods. The meta-optimization problem over gradient-based methods can be framed as a feedback control problem over the choice of hyperparameters, including the learning rate, momentum, and the preconditioner. Although the original optimal control problem is nonconvex, we show how recent methods from online nonstochastic control using convex relaxations can be used to circumvent the nonconvexity, and obtain regret guarantees vs. the best offline solution. This guarantees that in meta-optimization, given a sequence of optimization problems, we can learn a method that attains convergence comparable to that of the best optimization method in hindsight from a class of methods.

翻译：摘要：为特定优化实例（如学习率和动量）选择最佳超参数是一项重要但非凸的问题。因此，超梯度下降等迭代优化方法通常缺乏全局最优性保证。我们提出了一种用于数学优化的在线非随机控制方法论。首先，我们形式化了元优化的设定，即从一类方法中学习最优优化算法的在线学习表述。基于梯度方法的元优化问题可被建模为超参数（包括学习率、动量及预 conditioner）选择上的反馈控制问题。尽管原始最优控制问题是非凸的，我们展示了如何利用近期基于凸松弛的在线非随机控制方法绕过非凸性，并获得与最优离线解决方案相比的遗憾界。这保证了在元优化中，面对一系列优化问题，我们能够学习一种方法，使其收敛性能可媲美从一类方法中事后回顾得到的最优优化方法。