Skip connections and normalisation layers form two standard architectural components that are ubiquitous for the training of Deep Neural Networks (DNNs), but whose precise roles are poorly understood. Recent approaches such as Deep Kernel Shaping have made progress towards reducing our reliance on them, using insights from wide NN kernel theory to improve signal propagation in vanilla DNNs (which we define as networks without skips or normalisation). However, these approaches are incompatible with the self-attention layers present in transformers, whose kernels are intrinsically more complicated to analyse and control. And so the question remains: is it possible to train deep vanilla transformers? We answer this question in the affirmative by designing several approaches that use combinations of parameter initialisations, bias matrices and location-dependent rescaling to achieve faithful signal propagation in vanilla transformers. Our methods address various intricacies specific to signal propagation in transformers, including the interaction with positional encoding and causal masking. In experiments on WikiText-103 and C4, our approaches enable deep transformers without normalisation to train at speeds matching their standard counterparts, and deep vanilla transformers to reach the same performance as standard ones after about 5 times more iterations.

翻译：跳跃连接和归一化层是深度神经网络（DNNs）训练中普遍存在的两个标准架构组件，但其精确作用尚未被充分理解。近期诸如深度核整形等方法，利用宽神经网络核理论的分析洞见，已朝减少对这些组件的依赖取得进展——这些方法适用于无跳跃或归一化的朴素DNNs。然而，这类方法与Transformer中固有的自注意力层不兼容，因为后者的核函数在分析与调控上存在本质复杂性。因此问题依然存在：能否训练深度朴素Transformer？我们通过设计多种方法给出肯定回答，这些方法综合运用参数初始化、偏置矩阵和位置依赖重缩放，在朴素Transformer中实现忠实的信号传播。我们的方法解决了Transformer信号传播特有的若干技术细节，包括与位置编码和因果掩码的交互。在WikiText-103和C4数据集上的实验表明，我们的方法使无归一化的深度Transformer能以与标准模型相当的速度训练，而深度朴素Transformer仅需约5倍迭代次数即可达到与标准模型相同的性能。