In 1989 George Cybenko proved in a landmark paper that wide shallow neural networks can approximate arbitrary continuous functions on a compact set. This universal approximation theorem sparked a lot of follow-up research. Shen, Yang and Zhang determined optimal approximation rates for ReLU-networks in $L^p$-norms with $p \in [1,\infty)$. Kidger and Lyons proved a universal approximation theorem for deep narrow ReLU-networks. Telgarsky gave an example of a deep narrow ReLU-network that cannot be approximated by a wide shallow ReLU-network unless it has exponentially many neurons. However, there are even more questions that still remain unresolved. Are there any wide shallow ReLU-networks that cannot be approximated well by deep narrow ReLU-networks? Is the universal approximation theorem still true for other norms like the Sobolev norm $W^{1,1}$? Do these results hold for activation functions other than ReLU? We will answer all of those questions and more with a framework of two expressive powers. The first one is well-known and counts the maximal number of linear regions of a function calculated by a ReLU-network. We will improve the best known bounds for this expressive power. The second one is entirely new.

翻译：1989年，George Cybenko在一篇里程碑式论文中证明了宽浅神经网络能够在紧集上逼近任意连续函数。这一通用逼近定理引发了大量后续研究。Shen、Yang和Zhang确定了ReLU网络在$p \in [1,\infty)$的$L^p$范数下的最优逼近率。Kidger和Lyons证明了深窄ReLU网络的通用逼近定理。Telgarsky给出了一个实例，表明除非具有指数级数量的神经元，否则宽浅ReLU网络无法逼近该深窄ReLU网络。然而，仍有更多问题悬而未决：是否存在某些宽浅ReLU网络无法被深窄ReLU网络良好逼近？通用逼近定理对于Sobolev范数$W^{1,1}$等其他范数是否依然成立？这些结论对于ReLU以外的激活函数是否同样适用？我们将通过两个表达能力框架解答上述问题及更多疑问。第一个表达能力广为人知，它统计ReLU网络计算函数的线性区域最大数量。我们将改进该表达能力的最佳已知上界。第二个表达能力则完全是全新的。