We perform an effective-theory analysis of forward-backward signal propagation in wide and deep Transformers, i.e., residual neural networks with multi-head self-attention blocks and multilayer perceptron blocks. This analysis suggests particular width scalings of initialization and training hyperparameters for these models. We then take up such suggestions, training Vision and Language Transformers in practical setups.

翻译：我们对宽深Transformer（即具有多头自注意力块和多层感知机块的残差神经网络）中的前向-反向信号传播进行了有效理论分析。该分析为这些模型提出了初始化与训练超参数的具体宽度缩放建议。随后我们采纳这些建议，在实际配置中训练视觉与语言Transformer。