In this paper, we investigate the expressivity and approximation properties of deep neural networks employing the ReLU$^k$ activation function for $k \geq 2$. Although deep ReLU networks can approximate polynomials effectively, deep ReLU$^k$ networks have the capability to represent higher-degree polynomials precisely. Our initial contribution is a comprehensive, constructive proof for polynomial representation using deep ReLU$^k$ networks. This allows us to establish an upper bound on both the size and count of network parameters. Consequently, we are able to demonstrate a suboptimal approximation rate for functions from Sobolev spaces as well as for analytic functions. Additionally, through an exploration of the representation power of deep ReLU$^k$ networks for shallow networks, we reveal that deep ReLU$^k$ networks can approximate functions from a range of variation spaces, extending beyond those generated solely by the ReLU$^k$ activation function. This finding demonstrates the adaptability of deep ReLU$^k$ networks in approximating functions within various variation spaces.

翻译：本文研究了采用ReLU$^k$激活函数（$k \geq 2$）的深度神经网络的表达性与逼近性质。尽管深度ReLU网络能够有效逼近多项式，但深度ReLU$^k$网络具备精确表示高次多项式的独特能力。我们首先给出一个利用深度ReLU$^k$网络进行多项式表示的详尽构造性证明。基于此，我们建立了网络参数大小与数量的上界，进而证明了Sobolev空间函数及解析函数的次优逼近率。此外，通过探究深度ReLU$^k$网络对浅层网络的表示能力，我们揭示了深度ReLU$^k$网络能够逼近来自多种变差空间（包括不限于由ReLU$^k$激活函数生成的空间）的函数。这一发现展示了深度ReLU$^k$网络在不同变差空间中进行函数逼近的适应性。