Deep learning is also known as hierarchical learning, where the learner _learns_ to represent a complicated target function by decomposing it into a sequence of simpler functions to reduce sample and time complexity. This paper formally analyzes how multi-layer neural networks can perform such hierarchical learning _efficiently_ and _automatically_ by SGD on the training objective. On the conceptual side, we present a theoretical characterizations of how certain types of deep (i.e. super-constant layer) neural networks can still be sample and time efficiently trained on some hierarchical tasks, when no existing algorithm (including layerwise training, kernel method, etc) is known to be efficient. We establish a new principle called "backward feature correction", where the errors in the lower-level features can be automatically corrected when training together with the higher-level layers. We believe this is a key behind how deep learning is performing deep (hierarchical) learning, as opposed to layerwise learning or simulating some non-hierarchical method. On the technical side, we show for every input dimension $d > 0$, there is a concept class of degree $\omega(1)$ multi-variate polynomials so that, using $\omega(1)$-layer neural networks as learners, SGD can learn any function from this class in $\mathsf{poly}(d)$ time to any $\frac{1}{\mathsf{poly}(d)}$ error, through learning to represent it as a composition of $\omega(1)$ layers of quadratic functions using "backward feature correction." In contrast, we do not know any other simpler algorithm (including layerwise training, applying kernel method sequentially, training a two-layer network, etc) that can learn this concept class in $\mathsf{poly}(d)$ time even to any $d^{-0.01}$ error. As a side result, we prove $d^{\omega(1)}$ lower bounds for several non-hierarchical learners, including any kernel methods.

翻译：深度学习亦被称为层次化学习，其中学习器通过将复杂目标函数分解为一系列更简单的函数，以降低样本和时间复杂度，从而“学会”表示该函数。本文从理论角度分析多层神经网络如何通过随机梯度下降（SGD）在训练目标上高效且自动地实现此类层次化学习。在概念层面，我们揭示了特定类型的深度（即超常数层）神经网络如何在某些层次化任务中仍能实现样本和时间高效训练，而现有算法（包括逐层训练、核方法等）尚未被证明在该类任务上高效。我们提出一种名为“向后特征修正”的新原理，即低级特征中的误差可在与高级层共同训练时自动修正。我们认为，这是深度学习能够执行深度（层次化）学习的关键，而非采用逐层学习或模拟非层次化方法。在技术层面，我们证明：对于任意输入维度 $d > 0$，存在一个度为 $\omega(1)$ 的多变量多项式概念类，使得使用 $\omega(1)$ 层神经网络作为学习器时，SGD 可通过“向后特征修正”将该类中的任意函数学习表示为 $\omega(1)$ 层二次函数的复合，从而在 $\mathsf{poly}(d)$ 时间内使误差降至 $\frac{1}{\mathsf{poly}(d)}$。相比之下，目前未知任何更简单的算法（包括逐层训练、顺序应用核方法、训练两层网络等）能够在 $\mathsf{poly}(d)$ 时间内学习该概念类，即使将误差容忍度放宽至 $d^{-0.01}$ 亦不可行。作为附带结果，我们证明若干非层次化学习器（包括各种核方法）具有 $d^{\omega(1)}$ 的下界。