Let $\Omega = [0,1]^d$ be the unit cube in $\mathbb{R}^d$. We study the problem of how efficiently, in terms of the number of parameters, deep neural networks with the ReLU activation function can approximate functions in the Sobolev spaces $W^s(L_q(\Omega))$ and Besov spaces $B^s_r(L_q(\Omega))$, with error measured in the $L_p(\Omega)$ norm. This problem is important when studying the application of neural networks in a variety of fields, including scientific computing and signal processing, and has previously been solved only when $p=q=\infty$. Our contribution is to provide a complete solution for all $1\leq p,q\leq \infty$ and $s > 0$ for which the corresponding Sobolev or Besov space compactly embeds into $L_p$. The key technical tool is a novel bit-extraction technique which gives an optimal encoding of sparse vectors. This enables us to obtain sharp upper bounds in the non-linear regime where $p > q$. We also provide a novel method for deriving $L_p$-approximation lower bounds based upon VC-dimension when $p < \infty$. Our results show that very deep ReLU networks significantly outperform classical methods of approximation in terms of the number of parameters, but that this comes at the cost of parameters which are not encodable.

翻译：设$\Omega = [0,1]^d$为$\mathbb{R}^d$中的单位立方体。本文研究在参数数量约束下，使用ReLU激活函数的深度神经网络对Sobolev空间$W^s(L_q(\Omega))$与Besov空间$B^s_r(L_q(\Omega))$中函数的逼近效率问题，误差以$L_p(\Omega)$范数度量。该问题在科学计算与信号处理等神经网络应用领域具有重要意义，此前仅在$p=q=\infty$情形下得到解决。我们的贡献在于：对满足相应Sobolev或Besov空间紧嵌入$L_p$的所有$1\leq p,q\leq \infty$及$s>0$，给出了完整解。关键技术工具是一种新颖的比特提取技术，可实现对稀疏向量的最优编码，从而在$p>q$的非线性区域获得尖锐上界。此外，当$p<\infty$时，我们提出基于VC维的$L_p$-逼近下界推导新方法。结果表明：极深ReLU网络在参数数量上显著优于经典逼近方法，但代价是参数不可编码。