A fundamental open problem in deep learning theory is how to define and understand the stability of stochastic gradient descent (SGD) close to a fixed point. Conventional literature relies on the convergence of statistical moments, esp., the variance, of the parameters to quantify the stability. We revisit the definition of stability for SGD and use the \textit{convergence in probability} condition to define the \textit{probabilistic stability} of SGD. The proposed stability directly answers a fundamental question in deep learning theory: how SGD selects a meaningful solution for a neural network from an enormous number of solutions that may overfit badly. To achieve this, we show that only under the lens of probabilistic stability does SGD exhibit rich and practically relevant phases of learning, such as the phases of the complete loss of stability, incorrect learning, convergence to low-rank saddles, and correct learning. When applied to a neural network, these phase diagrams imply that SGD prefers low-rank saddles when the underlying gradient is noisy, thereby improving the learning performance. This result is in sharp contrast to the conventional wisdom that SGD prefers flatter minima to sharp ones, which we find insufficient to explain the experimental data. We also prove that the probabilistic stability of SGD can be quantified by the Lyapunov exponents of the SGD dynamics, which can easily be measured in practice. Our work potentially opens a new venue for addressing the fundamental question of how the learning algorithm affects the learning outcome in deep learning.

翻译：深度学习理论中的一个基本开放问题是如何定义和理解随机梯度下降（SGD）在不动点附近的稳定性。现有文献通常依赖于参数统计矩（尤其是方差）的收敛性来量化稳定性。我们重新审视了SGD稳定性的定义，并利用**依概率收敛**条件提出了SGD的**概率稳定性**概念。该稳定性直接回答了深度学习理论中的一个根本问题：SGD如何从大量可能导致严重过拟合的解中为神经网络选择有意义的解。为此，我们证明只有在概率稳定性的视角下，SGD才会展现出丰富且具有实际意义的学习阶段，例如完全失去稳定性、错误学习、收敛到低秩鞍点以及正确学习等阶段。当应用于神经网络时，这些相图表明，当底层梯度存在噪声时，SGD更倾向于低秩鞍点，从而提升学习性能。这一结果与学界认为SGD偏好平坦最小值而非尖锐最小值的传统观点形成鲜明对比，我们发现后者不足以解释实验数据。我们还证明，SGD的概率稳定性可通过其动力学的李雅普诺夫指数量化，而该指数在实践中易于测量。我们的工作有望为解决“学习算法如何影响深度学习结果”这一根本问题开辟新的路径。