We investigate the expressive power of depth-2 bandlimited random neural networks. A random net is a neural network where the hidden layer parameters are frozen with random assignment, and only the output layer parameters are trained by loss minimization. Using random weights for a hidden layer is an effective method to avoid non-convex optimization in standard gradient descent learning. It has also been adopted in recent deep learning theories. Despite the well-known fact that a neural network is a universal approximator, in this study, we mathematically show that when hidden parameters are distributed in a bounded domain, the network may not achieve zero approximation error. In particular, we derive a new nontrivial approximation error lower bound. The proof utilizes the technique of ridgelet analysis, a harmonic analysis method designed for neural networks. This method is inspired by fundamental principles in classical signal processing, specifically the idea that signals with limited bandwidth may not always be able to perfectly recreate the original signal. We corroborate our theoretical results with various simulation studies, and generally, two main take-home messages are offered: (i) Not any distribution for selecting random weights is feasible to build a universal approximator; (ii) A suitable assignment of random weights exists but to some degree is associated with the complexity of the target function.

翻译：我们研究了深度为2的带限随机神经网络的表达能力。随机网络是一种隐藏层参数随机分配并冻结，仅通过损失最小化训练输出层参数的神经网络。在隐藏层使用随机权重是避免标准梯度下降学习中非凸优化的有效方法，该方法也被近期深度学习理论所采用。尽管神经网络是通用逼近器已是众所周知的事实，但本研究从数学上证明：当隐藏参数分布在有界域内时，网络可能无法实现零逼近误差。具体而言，我们推导出一个新的非平凡逼近误差下界。该证明利用了脊波分析技术，这是一种专为神经网络设计的调和分析方法。此方法受经典信号处理基本原理解发，具体体现了"有限带宽信号可能无法始终完美重构原始信号"的思想。我们通过多种仿真研究验证了理论结果，并得出两个主要结论：(i) 并非任意随机权重分布都能构建通用逼近器；(ii) 存在合适的随机权重分配方式，但其在一定程度上与目标函数的复杂度相关。