The statistical essence of the Transformer architecture has long remained elusive: Is it a universal approximator, or a neural network version of known computational algorithms? Through rigorous algebraic proof, we show that the latter better describes Transformer's basic nature: Ordinary Least Squares (OLS) is a special case of the single-layer Linear Transformer. Using the spectral decomposition of the empirical covariance matrix, we construct a specific parameter setting where the attention mechanism's forward pass becomes mathematically equivalent to the OLS closed-form projection. This means attention can solve the problem in one forward pass, not by iterating. Building upon this prototypical case, we further uncover a decoupled slow and fast memory mechanism within Transformers. Finally, the evolution from our established linear prototype to standard Transformers is discussed. This progression facilitates the transition of the Hopfield energy function from linear to exponential memory capacity, thereby establishing a clear continuity between modern deep architectures and classical statistical inference.

翻译：Transformer架构的统计本质长期以来一直难以捉摸：它是一种通用逼近器，还是已知计算算法的神经网络版本？通过严格的代数证明，我们表明后者更能描述Transformer的基本性质：普通最小二乘法（OLS）是单层线性Transformer的一个特例。利用经验协方差矩阵的谱分解，我们构造了一个特定的参数设置，使得注意力机制的前向传播在数学上等价于OLS的闭式投影。这意味着注意力可以在一次前向传播中解决问题，而非通过迭代。以此原型案例为基础，我们进一步揭示了Transformer内部的解耦式快慢记忆机制。最后，讨论了从所建立的线性原型到标准Transformer的演化过程。这一演进促使Hopfield能量函数从线性记忆容量过渡到指数记忆容量，从而在现代深度架构与经典统计推断之间建立了清晰的连续性。