This paper argues that DNNs implement a computational Occam's razor -- finding the `simplest' algorithm that fits the data -- and that this could explain their incredible and wide-ranging success over more traditional statistical methods. We start with the discovery that the set of real-valued function $f$ that can be $ε$-approximated with a binary circuit of size at most $cε^{-γ}$ becomes convex in the `Harder than Monte Carlo' (HTMC) regime, when $γ>2$, allowing for the definition of a HTMC norm on functions. In parallel one can define a complexity measure on the parameters of a ResNets (a weighted $\ell_1$ norm of the parameters), which induce a `ResNet norm' on functions. The HTMC and ResNet norms can then be related by an almost matching sandwich bound. Thus minimizing this ResNet norm is equivalent to finding a circuit that fits the data with an almost minimal number of nodes (within a power of 2 of being optimal). ResNets thus appear as an alternative model for computation of real functions, better adapted to the HTMC regime and its convexity.

翻译：本文论证深度神经网络实现了一种计算上的奥卡姆剃刀——寻找拟合数据的最“简单”算法——这或可解释其相较传统统计方法取得的惊人且广泛成功。我们首先发现，在“比蒙特卡洛更难”（HTMC）的范式下，当γ>2时，可用规模不超过cε^{-γ}的二进制电路进行ε-逼近的实值函数集合f具有凸性，从而可在函数上定义HTMC范数。与此同时，我们可定义残差网络参数（参数的加权ℓ₁范数）的一种复杂度度量，该度量在函数空间上诱导出“残差网络范数”。随后可通过近乎匹配的夹逼界建立HTMC范数与残差网络范数之间的关联。因此，最小化该残差网络范数等价于寻找一个以近乎最小节点数（在最优值的2的幂次范围内）拟合数据的电路。残差网络由此成为实函数计算的替代模型，更适应于HTMC范式及其凸性特征。