A syntactic language model (SLM) incrementally generates a sentence with its syntactic tree in a left-to-right manner. We present Generative Pretrained Structured Transformers (GPST), an unsupervised SLM at scale capable of being pre-trained from scratch on raw texts with high parallelism. GPST circumvents the limitations of previous SLMs such as relying on gold trees and sequential training. It consists of two components, a usual SLM supervised by a uni-directional language modeling loss, and an additional composition model, which induces syntactic parse trees and computes constituent representations, supervised by a bi-directional language modeling loss. We propose a representation surrogate to enable joint parallel training of the two models in a hard-EM fashion. We pre-train GPST on OpenWebText, a corpus with $9$ billion tokens, and demonstrate the superiority of GPST over GPT-2 with a comparable size in numerous tasks covering both language understanding and language generation. Meanwhile, GPST also significantly outperforms existing unsupervised SLMs on left-to-right grammar induction, while holding a substantial acceleration on training.

翻译：句法语言模型以从左到右的方式逐步生成句子及其句法树。我们提出了生成式预训练结构化Transformer（GPST），一种能够在大规模原始文本上以高度并行方式从零开始预训练的无监督句法语言模型。GPST克服了以往句法语言模型的局限性，例如依赖金标准句法树和顺序训练。它由两部分组成：一个由单向语言建模损失监督的常规句法语言模型，以及一个额外的组合模型，该模型诱导句法解析树并计算成分表示，由双向语言建模损失监督。我们提出了一种表示代理方法，以硬期望最大化方式实现两个模型的联合并行训练。我们在包含90亿个令牌的OpenWebText语料库上预训练GPST，并证明在涵盖语言理解与语言生成的众多任务中，与同等规模的GPT-2相比，GPST具有优越性。同时，GPST在从左到右的语法归纳任务上显著优于现有无监督句法语言模型，并实现了训练速度的实质性提升。