Deep networks are often considered to be more expressive than shallow ones in terms of approximation. Indeed, certain functions can be approximated by deep networks provably more efficiently than by shallow ones, however, no tractable algorithms are known for learning such deep models. Separately, a recent line of work has shown that deep networks trained with gradient descent may behave like (tractable) kernel methods in a certain over-parameterized regime, where the kernel is determined by the architecture and initialization, and this paper focuses on approximation for such kernels. We show that for ReLU activations, the kernels derived from deep fully-connected networks have essentially the same approximation properties as their shallow two-layer counterpart, namely the same eigenvalue decay for the corresponding integral operator. This highlights the limitations of the kernel framework for understanding the benefits of such deep architectures. Our main theoretical result relies on characterizing such eigenvalue decays through differentiability properties of the kernel function, which also easily applies to the study of other kernels defined on the sphere.

翻译：深层网络在近似方面往往被认为比浅层网络更清晰。 事实上,某些功能可以比浅层网络更高效地被深层网络所近似,然而,在学习这种深层模型方面,没有已知的可移植算法。 另外,最近的一项工作表明,受过梯度下降训练的深层网络在某种超分化制度中可能表现得像(可吸引的)内核方法,因为内核是由构造和初始化决定的,本文侧重于这类内核的近似。我们表明,对于RELU的激活,由深层完全连接网络产生的内核基本上具有与浅层对等的近似特性,即对相应整体操作者而言,其类值同样衰减。这凸显了内核框架在了解这种深层结构的惠益方面的局限性。我们的主要理论结果依赖于通过内核功能的可变性特性来定性这种脑值衰变。 对于RELU的激活,从深层完全连接网络中产生的内核也很容易适用于对球体上定义的其他内核的研究。