Learning rate warmup is a popular and practical technique in training large-scale deep neural networks. Despite the huge success in practice, the theoretical advantages of this strategy of gradually increasing the learning rate at the beginning of the training process have not been fully understood. To resolve this gap between theory and practice, we first propose a novel family of generalized smoothness assumptions, and validate its applicability both theoretically and empirically. Under the novel smoothness assumption, we study the convergence properties of gradient descent (GD) in both deterministic and stochastic settings. It is shown that learning rate warmup consistently accelerates GD, and GD with warmup can converge at most $\Theta(T)$ times faster than with a non-increasing learning rate schedule in some specific cases, providing insights into the benefits of this strategy from an optimization theory perspective.

翻译：学习率预热是训练大规模深度神经网络时一种流行且实用的技术。尽管在实践中取得了巨大成功，这种在训练初期逐步增加学习率策略的理论优势尚未得到充分理解。为弥合理论与实践之间的差距，我们首先提出了一类新颖的广义光滑性假设，并从理论和实证两方面验证了其适用性。在此新颖的光滑性假设下，我们研究了梯度下降（GD）在确定性和随机设置中的收敛特性。研究表明，学习率预热能持续加速GD收敛，且在特定情况下，采用预热策略的GD可比采用非递增学习率调度的方法快至多$\Theta(T)$倍收敛，这从优化理论视角揭示了该策略的优越性。