In this work, we investigate new activation functions for achieving arbitrary-accuracy Sobolev approximation by fixed-size neural networks. We first show that any function in $W^{2,\infty}((a,b)^d)$ can be approximated with arbitrary accuracy, measured in the $W^{1,\infty}$-norm, by a fixed-size neural network using the Elementary Universal Activation Function ($\mathrm{EUAF}$). To extend this result to $W^{s,\infty}((a,b)^d)$ for $s\in\mathbb{N}$, we introduce a smooth activation $\mathrm{DUAF}_{\infty}$ from the family of Differentiable Universal Activation Functions ($\mathrm{DUAF}_n$). We prove that any function in $W^{s,\infty}((a,b)^d)$ can be approximated with arbitrary accuracy in the $W^{s-1,\infty}$-norm by a fixed-size $\mathrm{DUAF}_{\infty}$-activated network. We further construct sigmoidal variants $\widetilde{\mathrm{DUAF}}_n$ and show that, for every $1\leq s\leq n$, fixed-size $\widetilde{\mathrm{DUAF}}_n$-activated networks still approximate any $f\in W^{s,\infty}((a,b)^d)$ with arbitrary accuracy in the $W^{s-1,\infty}$-norm. In all these results, the width and depth bounds are computed explicitly, and the proposed activations are elementary.

翻译：本文研究实现固定规模神经网络任意精度Sobolev逼近的新型激活函数。我们首先证明，定义在$W^{2,\infty}((a,b)^d)$空间中的任意函数均可通过使用基本通用激活函数（$\mathrm{EUAF}$）的固定规模神经网络，在$W^{1,\infty}$范数度量下以任意精度逼近。为将该结果推广至$W^{s,\infty}((a,b)^d)$（$s\in\mathbb{N}$），我们引入可微通用激活函数族（$\mathrm{DUAF}_n$）中的光滑激活函数$\mathrm{DUAF}_{\infty}$。我们证明，定义在$W^{s,\infty}((a,b)^d)$中的任意函数可通过固定规模的$\mathrm{DUAF}_{\infty}$激活网络，在$W^{s-1,\infty}$范数下达到任意精度逼近。进一步构建S型变体$\widetilde{\mathrm{DUAF}}_n$，并证明对任意$1\leq s\leq n$，固定规模的$\widetilde{\mathrm{DUAF}}_n$激活网络仍可在$W^{s-1,\infty}$范数下逼近任意$f\in W^{s,\infty}((a,b)^d)$，且精度任意。上述所有结果中，网络宽度与深度界限均被显式计算，且所提出的激活函数均为初等函数。