This paper explores the expressive power of deep neural networks for a diverse range of activation functions. An activation function set $\mathscr{A}$ is defined to encompass the majority of commonly used activation functions, such as $\mathtt{ReLU}$, $\mathtt{LeakyReLU}$, $\mathtt{ReLU}^2$, $\mathtt{ELU}$, $\mathtt{SELU}$, $\mathtt{Softplus}$, $\mathtt{GELU}$, $\mathtt{SiLU}$, $\mathtt{Swish}$, $\mathtt{Mish}$, $\mathtt{Sigmoid}$, $\mathtt{Tanh}$, $\mathtt{Arctan}$, $\mathtt{Softsign}$, $\mathtt{dSiLU}$, and $\mathtt{SRS}$. We demonstrate that for any activation function $\varrho\in \mathscr{A}$, a $\mathtt{ReLU}$ network of width $N$ and depth $L$ can be approximated to arbitrary precision by a $\varrho$-activated network of width $4N$ and depth $2L$ on any bounded set. This finding enables the extension of most approximation results achieved with $\mathtt{ReLU}$ networks to a wide variety of other activation functions, at the cost of slightly larger constants.

翻译：本文探究了深度神经网络在多样化激活函数下的表达能力。我们定义了一个激活函数集 $\mathscr{A}$，该集合涵盖了绝大多数常用激活函数，例如 $\mathtt{ReLU}$、$\mathtt{LeakyReLU}$、$\mathtt{ReLU}^2$、$\mathtt{ELU}$、$\mathtt{SELU}$、$\mathtt{Softplus}$、$\mathtt{GELU}$、$\mathtt{SiLU}$、$\mathtt{Swish}$、$\mathtt{Mish}$、$\mathtt{Sigmoid}$、$\mathtt{Tanh}$、$\mathtt{Arctan}$、$\mathtt{Softsign}$、$\mathtt{dSiLU}$ 和 $\mathtt{SRS}$。我们证明，对于任意激活函数 $\varrho\in \mathscr{A}$，在任意有界集上，一个宽度为 $N$、深度为 $L$ 的 $\mathtt{ReLU}$ 网络可以被一个宽度为 $4N$、深度为 $2L$ 的 $\varrho$ 激活网络以任意精度逼近。这一发现使得利用 $\mathtt{ReLU}$ 网络获得的多数逼近结果能够以略微增大的常数为代价，推广至种类繁多的其他激活函数。