Let $\Omega = [0,1]^d$ be the unit cube in $\mathbb{R}^d$. We study the problem of how efficiently, in terms of the number of parameters, deep neural networks with the ReLU activation function can approximate functions in the Sobolev spaces $W^s(L_q(\Omega))$ and Besov spaces $B^s_r(L_q(\Omega))$, with error measured in the $L_p(\Omega)$ norm. This problem is important when studying the application of neural networks in a variety of fields, including scientific computing and signal processing, and has previously been completely solved only when $p=q=\infty$. Our contribution is to provide a complete solution for all $1\leq p,q\leq \infty$ and $s > 0$, including asymptotically matching upper and lower bounds. The key technical tool is a novel bit-extraction technique which gives an optimal encoding of sparse vectors. This enables us to obtain sharp upper bounds in the non-linear regime where $p > q$. We also provide a novel method for deriving $L_p$-approximation lower bounds based upon VC-dimension when $p < \infty$. Our results show that very deep ReLU networks significantly outperform classical methods of approximation in terms of the number of parameters, but that this comes at the cost of parameters which are not encodable.

翻译：设$\Omega = [0,1]^d$为$\mathbb{R}^d$中的单位立方体。我们研究以参数数量为衡量标准时，使用ReLU激活函数的深度神经网络对Sobolev空间$W^s(L_q(\Omega))$和Besov空间$B^s_r(L_q(\Omega))$中函数的逼近效率问题，误差以$L_p(\Omega)$范数度量。该问题对于研究神经网络在科学计算和信号处理等多个领域中的应用具有重要意义，且此前仅当$p=q=\infty$时获得完全解决。我们的贡献在于为所有$1\leq p,q\leq \infty$及$s>0$的情形提供完整解，包含渐近匹配的上界与下界。关键技术工具是一种新颖的位提取技术，可实现对稀疏向量的最优编码，使我们能够在$p>q$的非线性区域获得精确上界。此外，我们提出了一种基于VC维的新方法，用于推导$p<\infty$情形下的$L_p$逼近下界。研究结果表明，极深ReLU网络在参数数量方面显著优于经典逼近方法，但这一优势以参数不可编码为代价。