This paper presents Z-Code++, a new pre-trained language model optimized for abstractive text summarization. The model extends the state of the art encoder-decoder model using three techniques. First, we use a two-phase pre-training process to improve model's performance on low-resource summarization tasks. The model is first pre-trained using text corpora for language understanding, and then is continually pre-trained on summarization corpora for grounded text generation. Second, we replace self-attention layers in the encoder with disentangled attention layers, where each word is represented using two vectors that encode its content and position, respectively. Third, we use fusion-in-encoder, a simple yet effective method of encoding long sequences in a hierarchical manner. Z-Code++ creates new state of the art on 9 out of 13 text summarization tasks across 5 languages. Our model is parameter-efficient in that it outperforms the 600x larger PaLM-540B on XSum, and the finetuned 200x larger GPT3-175B on SAMSum. In zero-shot and few-shot settings, our model substantially outperforms the competing models.

翻译：本文提出Z-Code++，一种针对抽象式文本摘要优化的新型预训练语言模型。该模型通过三种技术扩展了最先进的编码器-解码器架构。首先，我们采用两阶段预训练流程提升模型在低资源摘要任务上的性能：模型首先使用文本语料库进行语言理解预训练，随后持续在摘要语料库上进行基于文本生成的有监督预训练。其次，我们使用解耦注意力层替代编码器中的自注意力层，其中每个单词分别通过内容向量和位置向量进行表示。第三，我们采用编码器融合技术，这是一种以分层方式编码长序列的简单有效方法。Z-Code++在覆盖5种语言的13项文本摘要任务中，有9项达到最新最优性能。该模型具有参数高效性：在XSum数据集上优于规模大600倍的PaLM-540B，在SAMSum数据集上优于微调后规模大200倍的GPT3-175B。在零样本和少样本设定下，该模型显著优于对比模型。