Deep learning methods - consisting of a class of deep neural networks (DNNs) trained by a stochastic gradient descent (SGD) optimization method - are nowadays key tools to solve data driven supervised learning problems. Despite the great success of SGD methods in the training of DNNs, it remains a fundamental open problem of research to explain the success and the limitations of such methods in rigorous theoretical terms. In particular, even in the standard setup of data driven supervised learning problems, it remained an open research problem to prove (or disprove) that SGD methods converge in the training of DNNs with the popular rectified linear unit (ReLU) activation function with high probability to global minimizers in the optimization landscape. In this work we answer this question negatively. Specifically, in this work we prove for a large class of SGD methods that the considered optimizer does with high probability not converge to global minimizers of the optimization problem. It turns out that the probability to not converge to a global minimizer converges at least exponentially quickly to one as the width of the first hidden layer of the ANN and the depth of the ANN, respectively, increase. The general non-convergence results of this work do not only apply to the plain vanilla standard SGD method but also to a large class of accelerated and adaptive SGD methods such as the momentum SGD, the Nesterov accelerated SGD, the Adagrad, the RMSProp, the Adam, the Adamax, the AMSGrad, and the Nadam optimizers.

翻译：深度学习方法——即通过随机梯度下降优化方法训练的一类深度神经网络——如今已成为解决数据驱动监督学习问题的关键工具。尽管SGD方法在训练DNNs方面取得了巨大成功，但如何用严格的理论术语解释此类方法的成功与局限性，仍然是基础性开放研究难题。特别是在数据驱动监督学习问题的标准框架下，证明（或证伪）SGD方法在训练使用流行的修正线性单元激活函数的DNNs时能以高概率收敛至优化景观中的全局极小值，始终是悬而未决的研究课题。本研究对此问题给出了否定答案。具体而言，我们针对一大类SGD方法证明：所考虑的优化器以高概率不收敛至优化问题的全局极小值。结果表明，不收敛至全局极小值的概率随着ANN第一个隐藏层宽度和ANN深度的增加分别以至少指数级速度趋近于一。本研究的普适非收敛性结论不仅适用于原始标准SGD方法，也适用于包括动量SGD、Nesterov加速SGD、Adagrad、RMSProp、Adam、Adamax、AMSGrad及Nadam优化器在内的一大类加速与自适应SGD方法。