Performance of optimization on quadratic problems sensitively depends on the low-lying part of the spectrum. For large (effectively infinite-dimensional) problems, this part of the spectrum can often be naturally represented or approximated by power law distributions, resulting in power law convergence rates for iterative solutions of these problems by gradient-based algorithms. In this paper, we propose a new spectral condition providing tighter upper bounds for problems with power law optimization trajectories. We use this condition to build a complete picture of upper and lower bounds for a wide range of optimization algorithms -- Gradient Descent, Steepest Descent, Heavy Ball, and Conjugate Gradients -- with an emphasis on the underlying schedules of learning rate and momentum. In particular, we demonstrate how an optimally accelerated method, its schedule, and convergence upper bound can be obtained in a unified manner for a given shape of the spectrum. Also, we provide first proofs of tight lower bounds for convergence rates of Steepest Descent and Conjugate Gradients under spectral power laws with general exponents. Our experiments show that the obtained convergence bounds and acceleration strategies are not only relevant for exactly quadratic optimization problems, but also fairly accurate when applied to the training of neural networks.

翻译：优化在二次问题上的性能敏感地依赖于谱的低端部分。对于大规模（实际上是无限维）问题，谱的这一部分通常可以用幂律分布自然地表示或近似，导致基于梯度的算法迭代求解这些问题时产生幂律收敛速率。在本文中，我们提出了一种新的谱条件，为具有幂律优化轨迹的问题提供了更紧的上界。我们利用该条件构建了针对多种优化算法（包括梯度下降、最速下降、重球法和共轭梯度法）的上下界完整图景，重点关注学习率和动量的基础调度策略。特别地，我们展示了如何针对给定的谱形状以统一方式获得最优加速方法、其调度策略及收敛上界。此外，我们首次给出了在一般指数幂律谱下最速下降和共轭梯度法收敛速率的紧下界证明。实验表明，所获得的收敛界和加速策略不仅适用于精确的二次优化问题，在应用于神经网络训练时也相当准确。