Stacking many layers to create truly deep neural networks is arguably what has led to the recent explosion of these methods. However, many properties of deep neural networks are not yet understood. One such mystery is the depth degeneracy phenomenon: the deeper you make your network, the closer your network is to a constant function on initialization. In this paper, we examine the evolution of the angle between two inputs to a ReLU neural network as a function of the number of layers. By using combinatorial expansions, we find precise formulas for how fast this angle goes to zero as depth increases. Our formulas capture microscopic fluctuations that are not visible in the popular framework of infinite width limits, and yet have a significant effect on predicted behaviour. The formulas are given in terms of the mixed moments of correlated Gaussians passed through the ReLU function. We also find a surprising combinatorial connection between these mixed moments and the Bessel numbers.

翻译：堆叠多层以构建真正深度神经网络的做法，无疑是近年来这些方法爆发式发展的关键。然而，深度神经网络的许多特性仍未得到充分理解。其中一个谜团便是深度退化现象：网络层数越深，其初始化后的输出就越接近常数函数。本文研究了ReLU神经网络中两个输入之间夹角随层数变化的演化规律。通过利用组合展开方法，我们推导出该夹角随深度增加而趋于零的精确收敛速率公式。这些公式捕捉到了在流行的无穷宽度极限框架下不可见的微观波动，而这些波动对预测行为具有显著影响。公式以通过ReLU函数的高斯相关变量的混合矩形式给出。此外，我们还发现这些混合矩与贝塞尔数之间存在令人惊奇的组合学关联。