Transformers excel empirically on tasks that process well-formed inputs according to some grammar, such as natural language and code. However, it remains unclear how they can process grammatical syntax. In fact, under standard complexity conjectures, standard transformers cannot recognize context-free languages (CFLs), a canonical formalism to describe syntax, or even regular languages, a subclass of CFLs. Past work has shown that $\mathcal{O}(\log(N))$ looping layers (w.r.t. input length $N$) allow transformers to recognize regular languages, but the question of context-free recognition with looped transformers remained open. In this work, we show that looped transformers with $\mathcal{O}(\log(N))$ looping layers and $\mathcal{O}(N^6)$ padding symbols can recognize all CFLs. However, training and inference with $\mathcal{O}(N^6)$ padding symbols is potentially impractical. Fortunately, we show that, for natural subclasses such as unambiguous CFLs, the recognition problem on transformers becomes more tractable, requiring $\mathcal{O}(N^3)$ padding. Empirically, looped and padded transformers perform better than fixed-depth transformers in recognizing CFLs. Overall, our results shed light on the intricacy of CFL recognition by transformers: while general recognition may require an intractable amount of padding, natural constraints such as unambiguity yield efficient recognition algorithms.

翻译：Transformer在处理符合语法规则的结构化输入（如自然语言和代码）时展现出卓越的实验性能。然而，它们如何解析语法结构仍不明确。事实上，在标准复杂度猜想下，标准Transformer既无法识别上下文无关语言（语法描述的经典形式化体系），也无法识别其子类正则语言。已有研究表明，$\mathcal{O}(\log(N))$个循环层（相对于输入长度$N$）可使Transformer识别正则语言，但循环Transformer是否能识别上下文无关语言仍是悬而未决的问题。本文证明，具有$\mathcal{O}(\log(N))$个循环层及$\mathcal{O}(N^6)$个填充符号的循环Transformer可识别所有上下文无关语言。但$\mathcal{O}(N^6)$量级的填充符号在训练与推理中可能缺乏实用性。值得关注的是，对于无歧义上下文无关语言等自然子类，Transformer的识别问题更易处理，仅需$\mathcal{O}(N^3)$个填充符号。实验表明，循环填充Transformer在上下文无关语言识别任务上优于固定深度Transformer。总体而言，我们的研究揭示了Transformer识别上下文无关语言的复杂性：虽然通用识别可能需要难以承受的填充量，但无歧义性等自然约束可催生高效的识别算法。