This paper provides norm-based generalization bounds for the Transformer architecture that do not depend on the input sequence length. We employ a covering number based approach to prove our bounds. We use three novel covering number bounds for the function class of bounded linear transformations to upper bound the Rademacher complexity of the Transformer. Furthermore, we show this generalization bound applies to the common Transformer training technique of masking and then predicting the masked word. We also run a simulated study on a sparse majority data set that empirically validates our theoretical findings.

翻译：本文为Transformer架构提供了与输入序列长度无关的范数泛化界。我们采用基于覆盖数的证明方法推导该界。通过提出有界线性变换函数类的三个新型覆盖数界，我们实现了对Transformer拉德马赫复杂度的上界估计。进一步地，我们证明该泛化界适用于掩码预测这一常见Transformer训练技术。最后，我们在稀疏多数数据集上开展仿真实验，实证验证了理论结果的有效性。