In this paper, we examine the long-run distribution of stochastic gradient descent (SGD) in general, non-convex problems. Specifically, we seek to understand which regions of the problem's state space are more likely to be visited by SGD, and by how much. Using an approach based on the theory of large deviations and randomly perturbed dynamical systems, we show that the long-run distribution of SGD resembles the Boltzmann-Gibbs distribution of equilibrium thermodynamics with temperature equal to the method's step-size and energy levels determined by the problem's objective and the statistics of the noise. In particular, we show that, in the long run, (a) the problem's critical region is visited exponentially more often than any non-critical region; (b) the iterates of SGD are exponentially concentrated around the problem's minimum energy state (which does not always coincide with the global minimum of the objective); (c) all other connected components of critical points are visited with frequency that is exponentially proportional to their energy level; and, finally (d) any component of local maximizers or saddle points is "dominated" by a component of local minimizers which is visited exponentially more often.

翻译：本文探讨了随机梯度下降（SGD）在一般非凸问题中的长期分布特性。具体而言，我们旨在理解SGD更可能访问问题状态空间中的哪些区域及其访问频率差异。基于大偏差理论与随机扰动动力系统的方法，我们证明SGD的长期分布类似于平衡态热力学中的玻尔兹曼-吉布斯分布，其中温度等于方法的步长，能量水平由问题目标函数与噪声统计特性共同决定。特别地，我们证明在长期运行中：（a）问题的临界区域被访问的概率呈指数级高于任何非临界区域；（b）SGD的迭代结果呈指数级集中在问题的最小能量状态附近（该状态并不总是与目标函数的全局最小值重合）；（c）所有其他连通分量临界点被访问的频率与其能量水平呈指数级正相关；最后（d）任何局部极大值点或鞍点分量都被局部极小值点分量所"支配"，后者被访问的概率呈指数级更高。