The current paradigm of large-scale pre-training and fine-tuning Transformer large language models has lead to significant improvements across the board in natural language processing. However, such large models are susceptible to overfitting to their training data, and as a result the models perform poorly when the domain changes. Also, due to the model's scale, the cost of fine-tuning the model to the new domain is large. Nonparametric Variational Information Bottleneck (NVIB) has been proposed as a regulariser for training cross-attention in Transformers, potentially addressing the overfitting problem. We extend the NVIB framework to replace all types of attention functions in Transformers, and show that existing pretrained Transformers can be reinterpreted as Nonparametric Variational (NV) models using a proposed identity initialisation. We then show that changing the initialisation introduces a novel, information-theoretic post-training regularisation in the attention mechanism, which improves out-of-domain generalisation without any training. This success supports the hypothesis that pretrained Transformers are implicitly NV Bayesian models.

翻译：当前大规模预训练和微调Transformer大型语言模型的范式显著提升了自然语言处理领域的全面性能。然而，此类大型模型容易过拟合其训练数据，导致在领域变化时表现欠佳。此外，受模型规模影响，将其微调至新领域的成本高昂。非参数变分信息瓶颈（NVIB）已被提出作为Transformer交叉注意力机制的正则化方法，有望解决过拟合问题。我们将NVIB框架扩展至替代Transformer中所有类型的注意力函数，并通过提出的恒等初始化方法，证明现有预训练Transformer可被重新解释为非参数变分（NV）模型。进一步研究表明，改变初始化方式会在注意力机制中引入一种新颖的、基于信息论的后训练正则化，该正则化无需任何训练即可提升域外泛化能力。这一成果支持了预训练Transformer本质上是隐式NV贝叶斯模型的假说。