Pre-trained language models (PLMs) have achieved remarkable success in natural language generation (NLG) tasks. Up to now, most NLG-oriented PLMs are pre-trained in an unsupervised manner using the large-scale general corpus. In the meanwhile, an increasing number of models pre-trained with labeled data (i.e. "supervised pre-training") showcase superior performance compared to unsupervised pre-trained models. Motivated by the success of supervised pre-training, we propose Multi-task superVised Pre-training (MVP) for natural language generation. We collect a large-scale natural language generation corpus, MVPCorpus, from $77$ datasets over $11$ diverse NLG tasks. Then we unify these examples into a general text-to-text format to pre-train the text generation model MVP in a supervised manner. For each task, we further pre-train specific soft prompts to stimulate the model's capacity to perform a specific task. Our MVP model can be seen as a practice that utilizes recent instruction tuning on relatively small PLMs. Extensive experiments have demonstrated the effectiveness and generality of our MVP model in a number of NLG tasks, which achieves state-of-the-art performance on $13$ out of $17$ datasets, outperforming BART by $9.3\%$ and Flan-T5 by $5.8\%$.

翻译：预训练语言模型（PLMs）在自然语言生成（NLG）任务中取得了显著成功。迄今为止，大多数面向NLG的PLMs均采用无监督方式在大型通用语料库上进行预训练。与此同时，越来越多的基于标注数据（即“监督预训练”）预训练的模型展现出优于无监督预训练模型的性能。受监督预训练成功的启发，我们提出了面向自然语言生成的多任务监督预训练（MVP）。我们从涵盖11个不同NLG任务的77个数据集中收集了一个大规模自然语言生成语料库MVPCorpus，然后将这些样本统一为通用的文本到文本格式，以监督方式预训练文本生成模型MVP。针对每个任务，我们进一步预训练特定的软提示（soft prompts），以激发模型执行特定任务的能力。我们的MVP模型可视为一种在相对较小的PLMs上应用近期指令微调技术的实践。大量实验证明了MVP模型在众多NLG任务中的有效性和泛化性，其在17个数据集中的13个上取得了最优性能，相较于BART提升9.3%，相较于Flan-T5提升5.8%。