The recipe behind the success of deep learning has been the combination of neural networks and gradient-based optimization. Understanding the behavior of gradient descent however, and particularly its instability, has lagged behind its empirical success. To add to the theoretical tools available to study gradient descent we propose the principal flow (PF), a continuous time flow that approximates gradient descent dynamics. To our knowledge, the PF is the only continuous flow that captures the divergent and oscillatory behaviors of gradient descent, including escaping local minima and saddle points. Through its dependence on the eigendecomposition of the Hessian the PF sheds light on the recently observed edge of stability phenomena in deep learning. Using our new understanding of instability we propose a learning rate adaptation method which enables us to control the trade-off between training stability and test set evaluation performance.

翻译：深度学习成功的秘诀在于神经网络与基于梯度的优化方法的结合。然而，对梯度下降行为（尤其是其不稳定性）的理解却滞后于其实证成功。为扩充研究梯度下降行为的理论工具，我们提出了主流动（PF），一种近似梯度下降动力学的连续时间流。据我们所知，PF是唯一能够捕捉梯度下降发散与振荡行为（包括逃离局部极小值和鞍点）的连续流。通过依赖于Hessian矩阵的特征分解，PF揭示了最近在深度学习中观察到的稳定性边缘现象。基于对不稳定性的新理解，我们提出了一种学习率自适应方法，能够控制训练稳定性与测试集评估性能之间的权衡。