Transformer language models exhibit intelligent behaviors such as understanding natural language, recognizing patterns, acquiring knowledge, reasoning, planning, reflecting and using tools. This paper explores how their underlying mechanics give rise to intelligent behaviors. We adopt a systems approach to analyze Transformers in detail and develop a mathematical framework that frames their dynamics as movement through embedding space. This novel perspective provides a principled way of thinking about the problem and reveals important insights related to the emergence of intelligence: 1. At its core the Transformer is a Embedding Space walker, mapping intelligent behavior to trajectories in this vector space. 2. At each step of the walk, it composes context into a single composite vector whose location in Embedding Space defines the next step. 3. No learning actually occurs during decoding; in-context learning and generalization are simply the result of different contexts composing into different vectors. 4. Ultimately the knowledge, intelligence and skills exhibited by the model are embodied in the organization of vectors in Embedding Space rather than in specific neurons or layers. These abilities are properties of this organization. 5. Attention's contribution boils down to the association-bias it lends to vector composition and which influences the aforementioned organization. However, more investigation is needed to ascertain its significance. 6. The entire model is composed from two principal operations: data independent filtering and data dependent aggregation. This generalization unifies Transformers with other sequence models and across modalities. Building upon this foundation we formalize and test a semantic space theory which posits that embedding vectors represent semantic concepts and find some evidence of its validity.

翻译：Transformer 语言模型展现出理解自然语言、识别模式、获取知识、推理、规划、反思和使用工具等智能行为。本文探讨了其底层机制如何催生这些智能行为。我们采用系统方法详细分析 Transformer，并发展了一个数学框架，将其动态过程描述为在嵌入空间中的运动。这一新颖视角为思考该问题提供了规范方法，并揭示了与智能涌现相关的重要见解：1. 其核心本质是嵌入空间中的行走者，将智能行为映射为该向量空间中的轨迹。2. 在每一步行走中，它将上下文组合成一个复合向量，该向量在嵌入空间中的位置决定了下一步的走向。3. 解码过程中实际上并未发生学习；上下文学习与泛化仅仅是不同上下文组合成不同向量的结果。4. 最终，模型所展现的知识、智能与技能体现在嵌入空间中向量的组织结构中，而非特定神经元或层中。这些能力是该组织结构的属性。5. 注意力机制的贡献归结于它为向量组合提供的关联偏向，这种偏向影响了前述的组织结构。然而，需要更多研究来确认其重要性。6. 整个模型由两个主要操作构成：数据无关的过滤与数据依赖的聚合。这一概括将 Transformer 与其他序列模型及不同模态统一起来。在此基础之上，我们形式化并测试了一种语义空间理论，该理论主张嵌入向量代表语义概念，并发现了一些支持其有效性的证据。