Transformers have theoretical limitations in modeling certain sequence-to-sequence tasks, yet it remains largely unclear if these limitations play a role in large-scale pretrained LLMs, or whether LLMs might effectively overcome these constraints in practice due to the scale of both the models themselves and their pretraining data. We explore how these architectural constraints manifest after pretraining, by studying a family of $\textit{retrieval}$ and $\textit{copying}$ tasks inspired by Liu et al. [2024a]. We use a recently proposed framework for studying length generalization [Huang et al., 2025] to provide guarantees for each of our settings. Empirically, we observe an $\textit{induction-versus-anti-induction}$ asymmetry, where pretrained models are better at retrieving tokens to the right (induction) rather than the left (anti-induction) of a query token. This asymmetry disappears upon targeted fine-tuning if length-generalization is guaranteed by theory. Mechanistic analysis reveals that this asymmetry is connected to the differences in the strength of induction versus anti-induction circuits within pretrained transformers. We validate our findings through practical experiments on real-world tasks demonstrating reliability risks. Our results highlight that pretraining selectively enhances certain transformer capabilities, but does not overcome fundamental length-generalization limits.

翻译：Transformer在建模某些序列到序列任务时存在理论局限性，然而目前尚不清楚这些局限性是否会影响大规模预训练的大型语言模型，或者LLMs是否可能凭借模型规模与预训练数据量的优势在实践中有效克服这些约束。我们通过研究受Liu等人[2024a]启发的$\textit{检索}$与$\textit{复制}$任务族，探索这些架构约束在预训练后的具体表现。我们采用Huang等人[2025]近期提出的长度泛化研究框架，为每个实验场景提供理论保证。实证研究发现存在$\textit{归纳与反归纳不对称性}$：预训练模型在检索查询标记右侧（归纳）而非左侧（反归纳）的标记时表现更优。若理论能保证长度泛化，这种不对称性在针对性微调后会消失。机理分析表明，该不对称性与预训练Transformer中归纳电路和反归纳电路的强度差异有关。我们通过在现实任务中的实践实验验证了研究结论，揭示了可靠性风险。研究结果强调：预训练会选择性增强Transformer的特定能力，但无法突破其根本的长度泛化限制。