It has been shown that deep neural networks of a large enough width are universal approximators but they are not if the width is too small. There were several attempts to characterize the minimum width $w_{\min}$ enabling the universal approximation property; however, only a few of them found the exact values. In this work, we show that the minimum width for $L^p$ approximation of $L^p$ functions from $[0,1]^{d_x}$ to $\mathbb R^{d_y}$ is exactly $\max\{d_x,d_y,2\}$ if an activation function is ReLU-Like (e.g., ReLU, GELU, Softplus). Compared to the known result for ReLU networks, $w_{\min}=\max\{d_x+1,d_y\}$ when the domain is $\smash{\mathbb R^{d_x}}$, our result first shows that approximation on a compact domain requires smaller width than on $\smash{\mathbb R^{d_x}}$. We next prove a lower bound on $w_{\min}$ for uniform approximation using general activation functions including ReLU: $w_{\min}\ge d_y+1$ if $d_x<d_y\le2d_x$. Together with our first result, this shows a dichotomy between $L^p$ and uniform approximations for general activation functions and input/output dimensions.

翻译：已有研究表明，足够宽的深度神经网络具有通用逼近性质，但宽度过小时则不具备。学界曾多次尝试刻画实现通用逼近的最小宽度$w_{\min}$，然而仅有少数研究给出了精确值。本文证明：若激活函数为类ReLU函数（如ReLU、GELU、Softplus），则从$[0,1]^{d_x}$到$\mathbb R^{d_y}$的$L^p$函数$L^p$逼近所需最小宽度恰为$\max\{d_x,d_y,2\}$。相较于已知结论——ReLU网络在定义域为$\smash{\mathbb R^{d_x}}$时$w_{\min}=\max\{d_x+1,d_y\}$——我们的结果首次表明，紧致域上的逼近所需宽度小于$\smash{\mathbb R^{d_x}}$上的情形。进一步地，我们针对包含ReLU在内的通用激活函数，证明了均匀逼近情形下$w_{\min}$的下界：当$d_x<d_y\le2d_x$时，有$w_{\min}\ge d_y+1$。结合第一个结果，这揭示了对于通用激活函数与输入/输出维度，$L^p$逼近与均匀逼近之间存在二分性。