Large Language Models (LLMs) have made significant strides in natural language processing, and a precise understanding of the internal mechanisms driving their success is essential. We regard LLMs as transforming embeddings via a discrete, coupled, nonlinear, dynamical system in high dimensions. This perspective motivates tracing the trajectories of individual tokens as they pass through transformer blocks, and linearizing the system along these trajectories through their Jacobian matrices. In our analysis of 38 openly available LLMs, we uncover the alignment of top left and right singular vectors of Residual Jacobians, as well as the emergence of linearity and layer-wise exponential growth. Notably, we discover that increased alignment $\textit{positively correlates}$ with model performance. Metrics evaluated post-training show significant improvement in comparison to measurements made with randomly initialized weights, highlighting the significant effects of training in transformers. These findings reveal a remarkable level of regularity that has previously been overlooked, reinforcing the dynamical interpretation and paving the way for deeper understanding and optimization of LLM architectures.

翻译：大型语言模型（LLMs）在自然语言处理领域取得了显著进展，而精确理解驱动其成功的内在机制至关重要。我们将LLMs视为高维空间中通过离散、耦合、非线性动力系统进行嵌入变换的过程。这一视角促使我们追踪单个词元在Transformer块中传递的轨迹，并通过雅可比矩阵沿这些轨迹对系统进行线性化。在对38个公开可用的LLM进行分析时，我们揭示了残差雅可比矩阵的左右奇异向量的对齐现象，以及线性特性和层间指数增长的出现。值得注意的是，我们发现增强的对齐性 $\textit{正向关联}$ 于模型性能。与随机初始化权重下的测量结果相比，训练后评估的指标显示出显著改善，这突显了训练对Transformer架构的重要影响。这些发现揭示了以往被忽视的高度规律性，强化了动力系统视角的解读，并为深入理解和优化LLM架构开辟了新途径。