Recent work describes what transformers can and cannot compute through connections to boolean circuits, but existing results lack exact characterizations and are sensitive to modeling choices. Padded transformers -- to whose input filler symbols such as ``...'' are appended -- emerge as a useful gadget for establishing equivalences to circuit classes by providing polynomial space for adaptive parallel computation. However, only a limited set of padded transformer idealizations has been studied, leaving open how robustly these equivalences hold under changes to attention type, model width, and uniformity. We find that, under practical assumptions, padded transformers are surprisingly robust to all of these, and identify numeric precision and model depth as the main factors affecting expressivity. Concretely, we prove that polynomially padded $\text{L-uniform}$ constant-precision transformers are equivalent to $\text{L-uniform AC}^0$, while growing-precision ones achieve $\text{L-uniform TC}^0$ regardless of width. Furthermore, looping enables sequential processing analogous to circuits: $\log^d N$-looped constant-precision transformers reach $\text{FO-uniform AC}^d$, and growing-precision ones reach $\text{FO-uniform TC}^d$. Interestingly, growing width or precision beyond logarithmic does not increase expressivity, and all our results hold for both softmax and average hard attention transformers.

翻译：近期研究通过将Transformer与布尔电路关联描述了其能计算和不能计算的内容，但现有结果缺乏精确刻画且对建模选择敏感。填充型Transformer——在其输入中附加如“...”等填充符号——通过为自适应并行计算提供多项式空间，成为建立与电路类等价关系的有用工具。然而，目前仅研究了有限几种填充型Transformer的理想化模型，尚不清楚这些等价关系在注意力类型、模型宽度和均匀性变化下的鲁棒性。我们发现，在实际假设下，填充型Transformer对所有这些变化表现出令人惊讶的鲁棒性，并确定了数值精度和模型深度是影响表达能力的主要因素。具体而言，我们证明多项式填充的$\text{L-uniform}$常数精度Transformer等价于$\text{L-uniform AC}^0$，而增长精度Transformer无论宽度如何都能达到$\text{L-uniform TC}^0$。此外，循环机制实现了类似于电路的顺序处理：$\log^d N$次循环的常数精度Transformer达到$\text{FO-uniform AC}^d$，增长精度Transformer达到$\text{FO-uniform TC}^d$。有趣的是，宽度或精度增长超过对数并不会提升表达能力，且我们所有结果对softmax和平均硬注意力Transformer均成立。