We investigate the geometric structure of learning dynamics in overparameterized transformer models through carefully controlled modular arithmetic tasks. Our primary finding is that despite operating in high-dimensional parameter spaces ($d=128$), transformer training trajectories rapidly collapse onto low-dimensional execution manifolds of dimension $3$--$4$. This dimensional collapse is robust across random seeds and moderate task difficulties, though the orientation of the manifold in parameter space varies between runs. We demonstrate that this geometric structure underlies several empirically observed phenomena: (1) sharp attention concentration emerges as saturation along routing coordinates within the execution manifold, (2) SGD commutators are preferentially aligned with the execution subspace (up to $10\times$ random baseline) early in training, with $>92\%$ of non-commutativity confined to orthogonal staging directions and this alignment decreasing as training converges, and (3) sparse autoencoders capture auxiliary routing structure but fail to isolate execution itself, which remains distributed across the low-dimensional manifold. Our results suggest a unifying geometric framework for understanding transformer learning, where the vast majority of parameters serve to absorb optimization interference while core computation occurs in a dramatically reduced subspace. These findings have implications for interpretability, training curriculum design, and understanding the role of overparameterization in neural network learning.

翻译：我们通过精心设计的模运算任务，研究过参数化Transformer模型学习动态的几何结构。主要发现是：尽管在高维参数空间（$d=128$）中运行，Transformer训练轨迹会迅速坍缩到维度为$3$--$4$的低维执行流形上。这种维度坍缩现象在不同随机种子和中等任务难度下均保持稳健，但流形在参数空间中的方向因训练轮次而异。我们证明这种几何结构是多个经验观测现象的基础：（1）尖锐的注意力集中现象表现为执行流形内路由坐标上的饱和；（2）训练早期，SGD对易子优先与执行子空间对齐（可达随机基线的$10\times$），超过$92\%$的非对易性被限制在正交的暂存方向上，且这种对齐程度随训练收敛而降低；（3）稀疏自编码器能够捕捉辅助路由结构，但未能隔离执行本身，后者仍分布在低维流形上。我们的结果表明，理解Transformer学习存在一个统一的几何框架：绝大多数参数用于吸收优化干扰，而核心计算发生在急剧缩减的子空间中。这些发现对可解释性、训练课程设计以及理解过参数化在神经网络学习中的作用具有重要意义。