Optimal Approximation Rates and Metric Entropy of ReLU$^k$ and Cosine Networks

This article addresses several fundamental issues associated with the approximation theory of neural networks, including the characterization of approximation spaces, the determination of the metric entropy of these spaces, and approximation rates of neural networks. For any activation function $\sigma$, we show that the largest Banach space of functions which can be efficiently approximated by the corresponding shallow neural networks is the space whose norm is given by the gauge of the closed convex hull of the set $\{\pm\sigma(\omega\cdot x + b)\}$. We characterize this space for the ReLU$^k$ and cosine activation functions and, in particular, show that the resulting gauge space is equivalent to the spectral Barron space if $\sigma=\cos$ and is equivalent to the Barron space when $\sigma={\rm ReLU}$. Our main result establishes the precise asymptotics of the $L^2$-metric entropy of the unit ball of these guage spaces and, as a consequence, the optimal approximation rates for shallow ReLU$^k$ networks. The sharpest previous results hold only in the special case that $k=0$ and $d=2$, where the metric entropy has been determined up to logarithmic factors. When $k > 0$ or $d > 2$, there is a significant gap between the previous best upper and lower bounds. We close all of these gaps and determine the precise asymptotics of the metric entropy for all $k \geq 0$ and $d\geq 2$, including removing the logarithmic factors previously mentioned. Finally, we use these results to quantify how much is lost by Barron's spectral condition relative to the convex hull of $\{\pm\sigma(\omega\cdot x + b)\}$ when $\sigma={\rm ReLU}^k$. Finally, we also show that the orthogonal greedy algorithm can algorithmically realize the improved approximation rates which have been derived.

翻译：此文章涉及与神经网络近似理论相关的若干基本问题, 包括近似空间的定性, 确定这些空间的公吨值, 以及神经网络的近似率。对于任何激活功能 $\ sgma$, 我们显示最大的 Banach 功能空间, 可以被相应的浅色神经网络有效近似, 其标准由设置 $\ pm\ sgma (\ omega\ cddd x + b) 的闭合 convex 柱体的仪表给定。我们给RU$ 的近似值和 cosine 激活功能设定了这个空间, 特别是, 如果 $\ gigma\ co$, 且当 $\ maqrq REU} 时, 最大Banchreach 空间的仪表空间相当于光谱, 美元 = $ rqrqrq rq ; 我们的主要结果确定, 美元美元美元 = 美元内基的单位球时, 美元数的值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值值