We study the approximation capacity of some variation spaces corresponding to shallow ReLU$^k$ neural networks. It is shown that sufficiently smooth functions are contained in these spaces with finite variation norms. For functions with less smoothness, the approximation rates in terms of the variation norm are established. Using these results, we are able to prove the optimal approximation rates in terms of the number of neurons for shallow ReLU$^k$ neural networks. It is also shown how these results can be used to derive approximation bounds for deep neural networks and convolutional neural networks (CNNs). As applications, we study convergence rates for nonparametric regression using three ReLU neural network models: shallow neural network, over-parameterized neural network, and CNN. In particular, we show that shallow neural networks can achieve the minimax optimal rates for learning H\"older functions, which complements recent results for deep neural networks. It is also proven that over-parameterized (deep or shallow) neural networks can achieve nearly optimal rates for nonparametric regression.

翻译：我们研究了与浅层ReLU$^k$神经网络对应的若干变差空间的逼近能力。结果表明，充分光滑的函数以有限变差范数包含于这些空间中。对于光滑性较弱的函数，建立了基于变差范数的逼近速率。利用这些结果，我们能够证明浅层ReLU$^k$神经网络关于神经元数目的最优逼近速率。同时展示了如何将这些结果用于推导深度神经网络和卷积神经网络（CNN）的逼近界。作为应用，我们研究了三种ReLU神经网络模型（浅层神经网络、过参数化神经网络和CNN）的非参数回归收敛速率。特别地，我们证明浅层神经网络能够达到学习Hölder函数的极小极大最优速率，这补充了近期关于深度神经网络的结果。此外，还证明了过参数化（深度或浅层）神经网络能够实现非参数回归的近优速率。