In this article, we consider convergence of stochastic gradient descent schemes (SGD), including momentum stochastic gradient descent (MSGD), under weak assumptions on the underlying landscape. More explicitly, we show that on the event that the SGD stays bounded we have convergence of the SGD if there is only a countable number of critical points or if the objective function satisfies Lojasiewicz-inequalities around all critical levels as all analytic functions do. In particular, we show that for neural networks with analytic activation function such as softplus, sigmoid and the hyperbolic tangent, SGD converges on the event of staying bounded, if the random variables modelling the signal and response in the training are compactly supported.

翻译：本文研究了随机梯度下降方案（SGD），包括动量随机梯度下降（MSGD），在较弱的地形假设条件下的收敛性。具体而言，我们证明：在SGD保持有界的事件中，若临界点数量可数，或目标函数在所有临界水平上满足Lojasiewicz不等式（所有解析函数均满足此类不等式），则SGD收敛。特别地，我们证明对于采用softplus、sigmoid和双曲正切等解析激活函数的神经网络，若训练过程中模拟信号和响应的随机变量具有紧支撑，则在保持有界的事件中SGD收敛。