We introduce a method for improving the structural understanding abilities of language models. Unlike previous approaches that finetune the models with task-specific augmentation, we pretrain language models on a collection of task-agnostic corpora to generate structures from text. Our structure pretraining enables zero-shot transfer of the learned knowledge that models have about the structure tasks. We study the performance of this approach on 28 datasets, spanning 10 structure prediction tasks including open information extraction, joint entity and relation extraction, named entity recognition, relation classification, semantic role labeling, event extraction, coreference resolution, factual probe, intent detection, and dialogue state tracking. We further enhance the pretraining with the task-specific training sets. We show that a 10B parameter language model transfers non-trivially to most tasks and obtains state-of-the-art performance on 21 of 28 datasets that we evaluate.

翻译：我们提出了一种提升语言模型结构理解能力的方法。与以往采用任务特定增强方式微调模型的方法不同，我们在任务无关的语料库集合上对语言模型进行预训练，使其能够从文本中生成结构。这种结构预训练实现了模型所学结构任务知识的零样本迁移。我们在涵盖开放信息抽取、联合实体关系抽取、命名实体识别、关系分类、语义角色标注、事件抽取、指代消解、事实探测、意图检测和对话状态追踪等10项结构预测任务的28个数据集上研究了该方法的性能。我们进一步利用任务特定训练集增强了预训练效果。研究表明，一个100亿参数的语言模型能够非平凡地迁移至大多数任务，并在所评估的28个数据集中的21个上取得了当前最优性能。