The recipe behind the success of deep learning has been the combination of neural networks and gradient-based optimization. Understanding the behavior of gradient descent however, and particularly its instability, has lagged behind its empirical success. To add to the theoretical tools available to study gradient descent we propose the principal flow (PF), a continuous time flow that approximates gradient descent dynamics. To our knowledge, the PF is the only continuous flow that captures the divergent and oscillatory behaviors of gradient descent, including escaping local minima and saddle points. Through its dependence on the eigendecomposition of the Hessian the PF sheds light on the recently observed edge of stability phenomena in deep learning. Using our new understanding of instability we propose a learning rate adaptation method which enables us to control the trade-off between training stability and test set evaluation performance.

翻译：深度学习成功的秘诀在于神经网络与基于梯度的优化方法的结合。然而，对梯度下降行为（特别是其不稳定性）的理解仍滞后于其经验性成功。为丰富研究梯度下降的理论工具，我们提出了主流动（PF）——一种逼近梯度下降动力学的连续时间流。据我们所知，PF是目前唯一能够捕捉梯度下降发散与振荡行为（包括逃离局部极小值和鞍点）的连续流。通过依赖Hessian矩阵的特征分解，PF揭示了近期在深度学习中观测到的“稳定性边缘”现象。基于对不稳定性的新认识，我们提出了一种学习率自适应方法，该方法能够调控训练稳定性与测试集评估性能之间的权衡关系。