Learning rate schedules used in practice bear little resemblance to those recommended by theory. We close much of this theory/practice gap, and as a consequence are able to derive new problem-adaptive learning rate schedules. Our key technical contribution is a refined analysis of learning rate schedules for a wide class of optimization algorithms (including SGD). In contrast to most prior works that study the convergence of the average iterate, we study the last iterate, which is what most people use in practice. When considering only worst-case analysis, our theory predicts that the best choice is the linear decay schedule: a popular choice in practice that sets the stepsize proportionally to $1 - t/T$, where $t$ is the current iteration and $T$ is the total number of steps. To go beyond this worst-case analysis, we use the observed gradient norms to derive schedules refined for any particular task. These refined schedules exhibit learning rate warm-up and rapid learning rate annealing near the end of training. Ours is the first systematic approach to automatically yield both of these properties. We perform the most comprehensive evaluation of learning rate schedules to date, evaluating across 10 diverse deep learning problems, a series of LLMs, and a suite of logistic regression problems. We validate that overall, the linear-decay schedule matches or outperforms all commonly used default schedules including cosine annealing, and that our schedule refinement method gives further improvements.

翻译：实践中使用的学习率调度策略与理论推荐的策略存在显著差异。我们大幅缩小了这一理论与实践的鸿沟，并据此推导出新的任务自适应学习率调度策略。我们的关键技术贡献在于对广泛优化算法（包括SGD）的学习率调度策略进行了精细化分析。与大多数关注平均迭代收敛的先前研究不同，我们聚焦于实际应用中人们最常使用的末次迭代。仅考虑最坏情况分析时，我们的理论预测线性衰减调度是最优选择：这恰好是实践中常用的策略，将步长设置为与$1 - t/T$成比例（其中$t$为当前迭代次数，$T$为总步数）。为突破最坏情况分析的局限，我们利用观测到的梯度范数为特定任务推导出精细化调度策略。这些精细化调度展现出学习率预热和训练末期快速学习率退火特性。这是首个系统性地自动生成这两种特性的方法。我们开展了迄今为止最全面的学习率调度策略评估，覆盖10个不同的深度学习问题、一系列大型语言模型及一组逻辑回归问题。实验验证表明：线性衰减调度整体上匹配或优于所有常用默认调度策略（包括余弦退火），而我们的调度精细化方法可带来进一步改进。