In this paper, we explore the structure of the penultimate Gram matrix in deep neural networks, which contains the pairwise inner products of outputs corresponding to a batch of inputs. In several architectures it has been observed that this Gram matrix becomes degenerate with depth at initialization, which dramatically slows training. Normalization layers, such as batch or layer normalization, play a pivotal role in preventing the rank collapse issue. Despite promising advances, the existing theoretical results do not extend to layer normalization, which is widely used in transformers, and can not quantitatively characterize the role of non-linear activations. To bridge this gap, we prove that layer normalization, in conjunction with activation layers, biases the Gram matrix of a multilayer perceptron towards the identity matrix at an exponential rate with depth at initialization. We quantify this rate using the Hermite expansion of the activation function.

翻译：本文探讨深度神经网络中倒数第二层Gram矩阵的结构，该矩阵包含一批输入对应输出之间的成对内积。在若干架构中观察到，该Gram矩阵在初始化时会随深度增加而退化，这极大拖慢了训练速度。归一化层（如批归一化或层归一化）在防止秩坍缩问题中起着关键作用。尽管已有令人瞩目的进展，现有理论结果无法推广到Transformer中广泛使用的层归一化，亦无法定量刻画非线性激活函数的作用。为弥合这一差距，我们证明：在初始化时，层归一化与激活层共同作用，会以随深度指数增长的速率使多层感知机的Gram矩阵偏向单位矩阵。我们利用激活函数的Hermite展开来量化该速率。