Recursion is a prominent feature of human language, and fundamentally challenging for self-attention due to the lack of an explicit recursive-state tracking mechanism. Consequently, Transformer language models poorly capture long-tail recursive structure and exhibit sample-inefficient syntactic generalization. This work introduces Pushdown Layers, a new self-attention layer that models recursive state via a stack tape that tracks estimated depths of every token in an incremental parse of the observed prefix. Transformer LMs with Pushdown Layers are syntactic language models that autoregressively and synchronously update this stack tape as they predict new tokens, in turn using the stack tape to softly modulate attention over tokens -- for instance, learning to "skip" over closed constituents. When trained on a corpus of strings annotated with silver constituency parses, Transformers equipped with Pushdown Layers achieve dramatically better and 3-5x more sample-efficient syntactic generalization, while maintaining similar perplexities. Pushdown Layers are a drop-in replacement for standard self-attention. We illustrate this by finetuning GPT2-medium with Pushdown Layers on an automatically parsed WikiText-103, leading to improvements on several GLUE text classification tasks.

翻译：递归是人类语言的一个显著特征，由于缺乏显式的递归状态跟踪机制，自注意力（self-attention）从根本上难以处理递归结构。因此，Transformer语言模型难以捕捉长尾递归结构，并表现出语法泛化的样本低效性。本文提出推叠层（Pushdown Layers），这是一种新型自注意力层，通过一个堆叠磁带（stack tape）来建模递归状态，该磁带跟踪观察到前缀的增量解析中每个标记的估计深度。配备推叠层的Transformer语言模型是句法语言模型，它们在预测新标记时自回归地同步更新这个堆叠磁带，并利用推叠层软性调节对标记的注意力——例如，学习“跳过”已闭合的句法成分。在使用银标准（silver）成分句法标记的字符串语料库上训练时，配备推叠层的Transformer在句法泛化上实现了显著提升，且样本效率提高3-5倍，同时保持相似的困惑度。推叠层可直接替代标准自注意力层。我们通过在自动解析的WikiText-103上使用推叠层微调GPT2-medium来证明这一点，从而在多个GLUE文本分类任务上取得改进。