We use the PAC-Bayesian theory for the setting of learning-to-optimize. To the best of our knowledge, we present the first framework to learn optimization algorithms with provable generalization guarantees (PAC-Bayesian bounds) and explicit trade-off between convergence guarantees and convergence speed, which contrasts with the typical worst-case analysis. Our learned optimization algorithms provably outperform related ones derived from a (deterministic) worst-case analysis. The results rely on PAC-Bayesian bounds for general, possibly unbounded loss-functions based on exponential families. Then, we reformulate the learning procedure into a one-dimensional minimization problem and study the possibility to find a global minimum. Furthermore, we provide a concrete algorithmic realization of the framework and new methodologies for learning-to-optimize, and we conduct four practically relevant experiments to support our theory. With this, we showcase that the provided learning framework yields optimization algorithms that provably outperform the state-of-the-art by orders of magnitude.

翻译：我们针对“学习优化”这一场景应用了PAC-Bayes理论。据我们所知，我们首次提出了一个框架，能够学习具有可证明泛化保证（PAC-Bayes界）的优化算法，并在收敛保证与收敛速度之间实现显式权衡，这与典型的基于最坏情况的分析截然不同。我们学习得到的优化算法可证明优于那些基于（确定性）最坏情况分析推导出的相关算法。这一结果依赖于针对基于指数族的一般（可能无界）损失函数的PAC-Bayes界。随后，我们将学习过程重新归结为一维最小化问题，并研究了寻找全局最小值的可能性。此外，我们为该框架提供了具体的算法实现方案以及学习优化的新方法，并通过四项具有实际相关性的实验来支撑我们的理论。借此，我们证明了所提供的学习框架能够产生可证明优于现有技术数个数量级的优化算法。