Classical worst-case optimization theory neither explains the success of optimization in machine learning, nor does it help with step size selection. We establish a connection between Bayesian Optimization (i.e. average case optimization theory) and classical optimization using a 'stochastic Taylor approximation' to rediscover gradient descent. This rediscovery yields a step size schedule we call Random Function Descent (RFD), which, in contrast to classical derivations, is scale invariant. Furthermore, our analysis of RFD step sizes yields a theoretical foundation for common step size heuristics such as gradient clipping and gradual learning rate warmup. We finally propose a statistical procedure for estimating the RFD step size schedule and validate this theory with a case study on the MNIST dataset.

翻译：经典的最坏情况优化理论既不能解释机器学习中优化方法的成功，也无法帮助选择步长。我们通过建立贝叶斯优化（即平均情况优化理论）与经典优化之间的联系，利用一种“随机泰勒近似”重新发现了梯度下降法。这一重新发现得到了一种步长调度方案，我们称之为随机函数下降（RFD），与经典推导不同，该方案具有尺度不变性。此外，我们对RFD步长的分析为常见的步长启发式方法（如梯度裁剪和渐进式学习率预热）提供了理论基础。最后，我们提出了一种用于估计RFD步长调度方案的统计方法，并通过在MNIST数据集上的案例研究验证了这一理论。