There is a growing literature on the study of large-width properties of deep Gaussian neural networks (NNs), i.e. deep NNs with Gaussian-distributed parameters or weights, and Gaussian stochastic processes. Motivated by some empirical and theoretical studies showing the potential of replacing Gaussian distributions with Stable distributions, namely distributions with heavy tails, in this paper we investigate large-width properties of deep Stable NNs, i.e. deep NNs with Stable-distributed parameters. For sub-linear activation functions, a recent work has characterized the infinitely wide limit of a suitable rescaled deep Stable NN in terms of a Stable stochastic process, both under the assumption of a ``joint growth" and under the assumption of a ``sequential growth" of the width over the NN's layers. Here, assuming a ``sequential growth" of the width, we extend such a characterization to a general class of activation functions, which includes sub-linear, asymptotically linear and super-linear functions. As a novelty with respect to previous works, our results rely on the use of a generalized central limit theorem for heavy tails distributions, which allows for an interesting unified treatment of infinitely wide limits for deep Stable NNs. Our study shows that the scaling of Stable NNs and the stability of their infinitely wide limits may depend on the choice of the activation function, bringing out a critical difference with respect to the Gaussian setting.

翻译：研究深度高斯神经网络（即参数或权重服从高斯分布的深度神经网络）与高斯随机过程的大宽度性质正涌现出大量文献。受实证与理论研究表明用稳定分布（即重尾分布）替代高斯分布具有潜力的启发，本文探究了深度稳定神经网络（即参数服从稳定分布的深度神经网络）的大宽度性质。针对亚线性激活函数，近期工作在“联合增长”与“序列增长”两种网络层宽度假设下，将经过适当重标的深层稳定神经网络的无限宽极限刻画为稳定随机过程。本文在宽度“序列增长”假设下，将该刻画推广至包含亚线性、渐近线性及超线性函数的通用激活函数类。相较于先前研究，本文创新性地采用重尾分布的广义中心极限定理，为深层稳定神经网络的无限宽极限提供了统一的优雅处理框架。研究表明，稳定神经网络的标度行为及其无限宽极限的稳定性可能依赖于激活函数的选择，这揭示了其与高斯情形之间的关键差异。