It is well-known that randomly initialized, push-forward, fully-connected neural networks weakly converge to isotropic Gaussian processes, in the limit where the width of all layers goes to infinity. In this paper, we propose to use the angular power spectrum of the limiting field to characterize the complexity of the network architecture. In particular, we define sequences of random variables associated with the angular power spectrum, and provide a full characterization of the network complexity in terms of the asymptotic distribution of these sequences as the depth diverges. On this basis, we classify neural networks as low-disorder, sparse, or high-disorder; we show how this classification highlights a number of distinct features for standard activation functions, and in particular, sparsity properties of ReLU networks. Our theoretical results are also validated by numerical simulations.

翻译：众所周知，在神经网络所有层宽度趋于无穷的极限情况下，随机初始化、前向传播的全连接神经网络会弱收敛于各向同性高斯过程。本文提出使用极限场的角功率谱来表征网络架构的复杂度。具体而言，我们定义了一系列与角功率谱相关的随机变量，并通过这些序列在深度趋于无穷时的渐近分布，完整刻画了网络的复杂度。在此基础上，我们将神经网络划分为低无序型、稀疏型和高无序型；我们展示了这一分类如何凸显标准激活函数的若干独特性质，特别是ReLU网络的稀疏特性。理论结果亦通过数值模拟得到了验证。