We investigate the expressive power of depth-2 bandlimited random neural networks. A random net is a neural network where the hidden layer parameters are frozen with random assignment, and only the output layer parameters are trained by loss minimization. Using random weights for a hidden layer is an effective method to avoid non-convex optimization in standard gradient descent learning. It has also been adopted in recent deep learning theories. Despite the well-known fact that a neural network is a universal approximator, in this study, we mathematically show that when hidden parameters are distributed in a bounded domain, the network may not achieve zero approximation error. In particular, we derive a new nontrivial approximation error lower bound. The proof utilizes the technique of ridgelet analysis, a harmonic analysis method designed for neural networks. This method is inspired by fundamental principles in classical signal processing, specifically the idea that signals with limited bandwidth may not always be able to perfectly recreate the original signal. We corroborate our theoretical results with various simulation studies, and generally, two main take-home messages are offered: (i) Not any distribution for selecting random weights is feasible to build a universal approximator; (ii) A suitable assignment of random weights exists but to some degree is associated with the complexity of the target function.

翻译：我们研究了深度为2的有限带宽随机神经网络的表达能力。随机网络是一种神经网络，其中隐藏层参数通过随机分配被冻结，仅输出层参数通过损失最小化进行训练。使用随机权重作为隐藏层是一种有效的方法，可避免标准梯度下降学习中的非凸优化问题。这一方法也已被近期深度学习理论所采纳。尽管神经网络是通用逼近器已是众所周知的事实，但在本研究中，我们通过数学证明，当隐藏参数分布于有界域时，网络可能无法实现零逼近误差。特别地，我们推导出了一个新的非平凡逼近误差下界。该证明利用了脊波分析技术——一种专为神经网络设计的调和分析方法。这一方法受到经典信号处理基本原理的启发，具体而言，源于有限带宽信号可能无法始终完美重构原始信号的观点。我们通过多种仿真研究验证了理论结果，并总体上提出了两个主要启示：（i）并非所有随机权重的选择分布都能构建出通用逼近器；（ii）存在合适的随机权重分配，但其在一定程度上与目标函数的复杂度相关。