In this paper, we provide a geometric interpretation of the structure of Deep Learning (DL) networks, characterized by $L$ hidden layers, a ReLU ramp activation function, an $\mathcal{L}^2$ Schatten class (or Hilbert-Schmidt) cost function, and input and output spaces $\mathbb{R}^Q$ with equal dimension $Q\geq1$. The hidden layers are also defined on $\mathbb{R}^{Q}$; the training input size $N$ can be arbitrarily large - thus, we are considering the underparametrized regime. We apply our recent results on shallow neural networks to construct an explicit family of minimizers for the global minimum of the cost function in the case $L\geq Q$, which we show to be degenerate. In the context presented here, the hidden layers of the DL network "curate" the training inputs by recursive application of a truncation map that minimizes the noise to signal ratio of the training inputs. Moreover, we determine a set of $2^Q-1$ distinct degenerate local minima of the cost function. Our constructions make no use of gradient descent algorithms at all.

翻译：本文给出了深度学习（DL）网络结构的几何解释，该网络由$L$个隐藏层、ReLU斜坡激活函数、$\mathcal{L}^2$ Schatten类（或Hilbert-Schmidt）代价函数以及维数相等（$Q\geq1$）的输入输出空间$\mathbb{R}^Q$构成。隐藏层同样定义在$\mathbb{R}^{Q}$上；训练样本数量$N$可任意大——因此我们考虑的是欠参数化情形。我们应用近期关于浅层神经网络的研究成果，在$L\geq Q$条件下构造了一族显式的全局代价函数极小化子，并证明该最小值是退化的。在本文框架下，DL网络的隐藏层通过递归应用一种最小化训练输入信噪比的截断映射，实现对训练输入的“筛选”。此外，我们确定了代价函数的一组$2^Q-1$个互异的退化局部极小值。本文所有构造均未使用任何梯度下降算法。