During pretraining, the Pre-LayerNorm transformer suffers from a gradient magnitude mismatch: gradients at early layers are much larger than at later layers. These issues can be alleviated by our proposed NormFormer architecture, which adds three normalization operations to each layer: a Layer Norm after self attention, head-wise scaling of self-attention outputs, and a Layer Norm after the first fully connected layer. The extra operations incur negligible compute cost (+0.4% parameter increase), but improve pretraining perplexity and downstream task performance for both causal and masked language models ranging from 125 Million to 2.7 Billion parameters. For example, adding NormFormer on top of our strongest 1.3B parameter baseline can reach equal perplexity 24% faster, or converge 0.27 perplexity better in the same compute budget. This model reaches GPT3-Large (1.3B) zero shot performance 60% faster. For masked language modeling, NormFormer improves fine-tuned GLUE performance by 1.9% on average. Code to train NormFormer models is available in fairseq https://github.com/pytorch/fairseq/tree/main/examples/normformer .

翻译：在预训练期间,LayerNorm前变压器存在一个梯度错差:早期层的梯度比后层的梯度要大得多。这些问题可以通过我们提议的NormFormer结构来缓解。 我们提议的NormFormer结构为每个层增加了三个正常化操作: 自我关注后的层圈, 自我注意后头视的输出尺度和第一个完全连接的层之后的层圈。 额外操作产生可忽略不计的计算成本( +0. 4% 参数增加), 但对于从125万到2. 7 10 亿的因果和遮蔽语言模型, 培训前的难度和下游任务性能都有所改善。 例如, 在我们1.3B 最强的参数基线之上添加NormFormer可以更快地达到相等的复度 24%, 在同一计算预算中将0. 27 的复度集中得更好。 这个模型达到 GPT3- Large (1.3B) 零射速性能60% 。 对于蒙语言模型来说, NormFormers 平均将精调GLUE的性表现提高1.9%。培训NormFormer模型的代码可以在 https://gistrebub/ report/ report/ report/formormormexx/ regres/ regreystrystry/