We present a Lipschitz continuous Transformer, called LipsFormer, to pursue training stability both theoretically and empirically for Transformer-based models. In contrast to previous practical tricks that address training instability by learning rate warmup, layer normalization, attention formulation, and weight initialization, we show that Lipschitz continuity is a more essential property to ensure training stability. In LipsFormer, we replace unstable Transformer component modules with Lipschitz continuous counterparts: CenterNorm instead of LayerNorm, spectral initialization instead of Xavier initialization, scaled cosine similarity attention instead of dot-product attention, and weighted residual shortcut. We prove that these introduced modules are Lipschitz continuous and derive an upper bound on the Lipschitz constant of LipsFormer. Our experiments show that LipsFormer allows stable training of deep Transformer architectures without the need of careful learning rate tuning such as warmup, yielding a faster convergence and better generalization. As a result, on the ImageNet 1K dataset, LipsFormer-Swin-Tiny based on Swin Transformer training for 300 epochs can obtain 82.7\% without any learning rate warmup. Moreover, LipsFormer-CSwin-Tiny, based on CSwin, training for 300 epochs achieves a top-1 accuracy of 83.5\% with 4.7G FLOPs and 24M parameters. The code will be released at \url{https://github.com/IDEA-Research/LipsFormer}.

翻译：我们提出了一种Lipschitz连续的Transformer模型——LipsFormer，旨在从理论和实践上提升基于Transformer模型的训练稳定性。与先前通过学习率预热、层归一化、注意力机制设计和权重初始化等实用技巧来解决训练不稳定性的方法不同，我们证明Lipschitz连续性是确保训练稳定性的更本质属性。在LipsFormer中，我们将不稳定的Transformer组件模块替换为Lipschitz连续对应模块：用CenterNorm替代LayerNorm、用谱初始化替代Xavier初始化、用缩放余弦相似度注意力替代点积注意力，并采用加权残差跳跃连接。我们证明了这些引入的模块具有Lipschitz连续性，并推导了LipsFormer的Lipschitz常数上界。实验表明，LipsFormer无需精细调整学习率（如预热）即可稳定训练深层Transformer架构，实现更快的收敛速度和更好的泛化性能。在ImageNet 1K数据集上，基于Swin Transformer训练的LipsFormer-Swin-Tiny在300轮训练中无需任何学习率预热即可达到82.7%的准确率。此外，基于CSwin的LipsFormer-CSwin-Tiny在300轮训练中以4.7G FLOPs和24M参数实现了83.5%的top-1准确率。代码将发布于\url{https://github.com/IDEA-Research/LipsFormer}。