This paper explores the expressive power of deep neural networks for a diverse range of activation functions. An activation function set $\mathscr{A}$ is defined to encompass the majority of commonly used activation functions, such as $\mathtt{ReLU}$, $\mathtt{LeakyReLU}$, $\mathtt{ReLU}^2$, $\mathtt{ELU}$, $\mathtt{SELU}$, $\mathtt{Softplus}$, $\mathtt{GELU}$, $\mathtt{SiLU}$, $\mathtt{Swish}$, $\mathtt{Mish}$, $\mathtt{Sigmoid}$, $\mathtt{Tanh}$, $\mathtt{Arctan}$, $\mathtt{Softsign}$, $\mathtt{dSiLU}$, and $\mathtt{SRS}$. We demonstrate that for any activation function $\varrho\in \mathscr{A}$, a $\mathtt{ReLU}$ network of width $N$ and depth $L$ can be approximated to arbitrary precision by a $\varrho$-activated network of width $3N$ and depth $2L$ on any bounded set. This finding enables the extension of most approximation results achieved with $\mathtt{ReLU}$ networks to a wide variety of other activation functions, albeit with slightly increased constants. Significantly, we establish that the (width,$\,$depth) scaling factors that appeared in the previous result can be further reduced from $(3,2)$ to $(1,1)$ if $\varrho$ falls within a specific subset of $\mathscr{A}$. This subset includes activation functions such as $\mathtt{ELU}$, $\mathtt{SELU}$, $\mathtt{Softplus}$, $\mathtt{GELU}$, $\mathtt{SiLU}$, $\mathtt{Swish}$, and $\mathtt{Mish}$.

翻译：本文探究了深度神经网络在多样化激活函数下的表达能力。我们定义了激活函数集合$\mathscr{A}$，涵盖大部分常用激活函数，如$\mathtt{ReLU}$、$\mathtt{LeakyReLU}$、$\mathtt{ReLU}^2$、$\mathtt{ELU}$、$\mathtt{SELU}$、$\mathtt{Softplus}$、$\mathtt{GELU}$、$\mathtt{SiLU}$、$\mathtt{Swish}$、$\mathtt{Mish}$、$\mathtt{Sigmoid}$、$\mathtt{Tanh}$、$\mathtt{Arctan}$、$\mathtt{Softsign}$、$\mathtt{dSiLU}$和$\mathtt{SRS}$。我们证明，对于任意激活函数$\varrho\in \mathscr{A}$，宽度为$N$、深度为$L$的$\mathtt{ReLU}$网络，可在任意有界集上被宽度为$3N$、深度为$2L$的$\varrho$激活网络以任意精度逼近。该发现使得大部分基于$\mathtt{ReLU}$网络的逼近结果得以推广至其他广泛激活函数，尽管常数略有增加。重要的是，我们确立：若$\varrho$属于$\mathscr{A}$的特定子集（包括$\mathtt{ELU}$、$\mathtt{SELU}$、$\mathtt{Softplus}$、$\mathtt{GELU}$、$\mathtt{SiLU}$、$\mathtt{Swish}$和$\mathtt{Mish}$等激活函数），前述结果中的（宽度，深度）缩放因子可从$(3,2)$进一步降低至$(1,1)$。