This paper explores the expressive power of deep neural networks for a diverse range of activation functions. An activation function set $\mathscr{A}$ is defined to encompass the majority of commonly used activation functions, such as $\mathtt{ReLU}$, $\mathtt{LeakyReLU}$, $\mathtt{ReLU}^2$, $\mathtt{ELU}$, $\mathtt{CELU}$, $\mathtt{SELU}$, $\mathtt{Softplus}$, $\mathtt{GELU}$, $\mathtt{SiLU}$, $\mathtt{Swish}$, $\mathtt{Mish}$, $\mathtt{Sigmoid}$, $\mathtt{Tanh}$, $\mathtt{Arctan}$, $\mathtt{Softsign}$, $\mathtt{dSiLU}$, and $\mathtt{SRS}$. We demonstrate that for any activation function $\varrho\in \mathscr{A}$, a $\mathtt{ReLU}$ network of width $N$ and depth $L$ can be approximated to arbitrary precision by a $\varrho$-activated network of width $3N$ and depth $2L$ on any bounded set. This finding enables the extension of most approximation results achieved with $\mathtt{ReLU}$ networks to a wide variety of other activation functions, albeit with slightly increased constants. Significantly, we establish that the (width,$\,$depth) scaling factors can be further reduced from $(3,2)$ to $(1,1)$ if $\varrho$ falls within a specific subset of $\mathscr{A}$. This subset includes activation functions such as $\mathtt{ELU}$, $\mathtt{CELU}$, $\mathtt{SELU}$, $\mathtt{Softplus}$, $\mathtt{GELU}$, $\mathtt{SiLU}$, $\mathtt{Swish}$, and $\mathtt{Mish}$.

翻译：本文探索了深度神经网络在多样化激活函数下的表达能力。我们定义了一个激活函数集$\mathscr{A}$，该集合涵盖了大多数常用激活函数，例如$\mathtt{ReLU}$、$\mathtt{LeakyReLU}$、$\mathtt{ReLU}^2$、$\mathtt{ELU}$、$\mathtt{CELU}$、$\mathtt{SELU}$、$\mathtt{Softplus}$、$\mathtt{GELU}$、$\mathtt{SiLU}$、$\mathtt{Swish}$、$\mathtt{Mish}$、$\mathtt{Sigmoid}$、$\mathtt{Tanh}$、$\mathtt{Arctan}$、$\mathtt{Softsign}$、$\mathtt{dSiLU}$和$\mathtt{SRS}$。我们证明，对于任意激活函数$\varrho\in \mathscr{A}$，一个宽度为$N$、深度为$L$的$\mathtt{ReLU}$网络，可以在任意有界集合上被一个宽度为$3N$、深度为$2L$的$\varrho$激活网络以任意精度逼近。这一发现使得大多数使用$\mathtt{ReLU}$网络获得的逼近结果能够扩展到多种其他激活函数，尽管常数略有增加。重要的是，我们确定如果$\varrho$属于$\mathscr{A}$的特定子集，则（宽度，深度）缩放因子可以从$(3,2)$进一步减小到$(1,1)$。该子集包括诸如$\mathtt{ELU}$、$\mathtt{CELU}$、$\mathtt{SELU}$、$\mathtt{Softplus}$、$\mathtt{GELU}$、$\mathtt{SiLU}$、$\mathtt{Swish}$和$\mathtt{Mish}$等激活函数。