We investigate the expressive power of depth-2 bandlimited random neural networks. A random net is a neural network where the hidden layer parameters are frozen with random assignment, and only the output layer parameters are trained by loss minimization. Using random weights for a hidden layer is an effective method to avoid non-convex optimization in standard gradient descent learning. It has also been adopted in recent deep learning theories. Despite the well-known fact that a neural network is a universal approximator, in this study, we mathematically show that when hidden parameters are distributed in a bounded domain, the network may not achieve zero approximation error. In particular, we derive a new nontrivial approximation error lower bound. The proof utilizes the technique of ridgelet analysis, a harmonic analysis method designed for neural networks. This method is inspired by fundamental principles in classical signal processing, specifically the idea that signals with limited bandwidth may not always be able to perfectly recreate the original signal. We corroborate our theoretical results with various simulation studies, and generally, two main take-home messages are offered: (i) Not any distribution for selecting random weights is feasible to build a universal approximator; (ii) A suitable assignment of random weights exists but to some degree is associated with the complexity of the target function.

翻译：我们研究了深度为2的带限随机神经网络的表达能力。随机网络是一种隐藏层参数被随机冻结、仅通过损失最小化训练输出层参数的神经网络。使用随机权重初始化隐藏层是避免标准梯度下降中非凸优化的有效方法，该方法近年来也被广泛应用于深度学习理论研究。尽管神经网络是通用逼近器早已成为共识，但本研究从数学上证明：当隐藏参数分布在有界域中时，网络可能无法实现零逼近误差。我们特别推导了一个新的非平凡逼近误差下界。该证明利用了脊波分析技术——一种针对神经网络设计的调和分析方法，其思想源自经典信号处理中的基本原理：有限带宽信号可能无法始终完美重建原始信号。我们通过多种仿真实验验证了理论结果，并得出两个主要结论：(i) 并非任意随机权重分布都能构建通用逼近器；(ii) 存在合适的随机权重分配，但其效果在一定程度上取决于目标函数的复杂度。