We investigate properties of neural networks that use both ReLU and $x^2$ as activation functions and build upon previous results to show that both analytic functions and functions in Sobolev spaces can be approximated by such networks of constant depth to arbitrary accuracy, demonstrating optimal order approximation rates across all nonlinear approximators, including standard ReLU networks. We then show how to leverage low local dimensionality in some contexts to overcome the curse of dimensionality, obtaining approximation rates that are optimal for unknown lower-dimensional subspaces.

翻译：我们研究了同时使用ReLU和$x^2$作为激活函数的神经网络的性质，并基于先前结果证明：解析函数以及Sobolev空间中的函数均可被这种恒定深度的网络以任意精度逼近，展现了包括标准ReLU网络在内的所有非线性逼近器中的最优阶逼近速率。随后，我们展示了如何在某些场景中利用低局部维度来克服维度灾难，从而获得针对未知低维子空间的最优逼近速率。