We study the training of deep neural networks by gradient descent where floating-point arithmetic is used to compute the gradients. In this framework and under realistic assumptions, we demonstrate that it is highly unlikely to find ReLU neural networks that maintain, in the course of training with gradient descent, superlinearly many affine pieces with respect to their number of layers. In virtually all approximation theoretical arguments that yield high-order polynomial rates of approximation, sequences of ReLU neural networks with exponentially many affine pieces compared to their numbers of layers are used. As a consequence, we conclude that approximating sequences of ReLU neural networks resulting from gradient descent in practice differ substantially from theoretically constructed sequences. The assumptions and the theoretical results are compared to a numerical study, which yields concurring results.

翻译：我们研究使用浮点算术计算梯度时，通过梯度下降训练深度神经网络的过程。在此框架下，基于现实假设，我们证明在梯度下降训练过程中，很难找到其仿射片段数量相对于层数保持超线性增长的ReLU神经网络。在几乎所有能实现高阶多项式逼近率的逼近理论论证中，都使用了仿射片段数量相对于层数呈指数增长的ReLU神经网络序列。因此，我们得出结论：实践中由梯度下降产生的ReLU神经网络逼近序列与理论构建的序列存在本质差异。我们将相关假设和理论结果与数值实验进行对比，二者结论一致。