Large-scale transformer models have shown remarkable performance in language modelling tasks. However, such models feature billions of parameters, leading to difficulties in their deployment and prohibitive training costs from scratch. To reduce the number of the parameters in the GPT-2 architecture, we replace the matrices of fully-connected layers with the corresponding Tensor Train Matrix~(TTM) structure. Finally, we customize forward and backward operations through the TTM-based layer for simplicity and the stableness of further training. % The resulting GPT-2-based model stores up to 40% fewer parameters, showing the perplexity comparable to the original model. On the downstream tasks, including language understanding and text summarization, the model performs similarly to the original GPT-2 model. The proposed tensorized layers could be used to efficiently pre-training other Transformer models.

翻译：大规模Transformer模型在语言建模任务中表现出卓越性能。然而，此类模型包含数十亿参数，导致部署困难且从头开始的训练成本过高。为减少GPT-2架构中的参数量，我们采用张量列车矩阵（TTM）结构替代全连接层的矩阵。最终，我们针对基于TTM的层定制了前向与反向运算，以简化操作并确保后续训练的稳定性。所提出的基于GPT-2的模型可减少高达40%的参数存储，其困惑度与原始模型相当。在包括语言理解和文本摘要等下游任务中，该模型表现与原始GPT-2模型相近。所提出的张量化层可有效用于其他Transformer模型的预训练。