Transformers have significantly advanced the field of natural language processing, but comprehending their internal mechanisms remains a challenge. In this paper, we introduce a novel geometric perspective that elucidates the inner mechanisms of transformer operations. Our primary contribution is illustrating how layer normalization confines the latent features to a hyper-sphere, subsequently enabling attention to mold the semantic representation of words on this surface. This geometric viewpoint seamlessly connects established properties such as iterative refinement and contextual embeddings. We validate our insights by probing a pre-trained 124M parameter GPT-2 model. Our findings reveal clear query-key attention patterns in early layers and build upon prior observations regarding the subject-specific nature of attention heads at deeper layers. Harnessing these geometric insights, we present an intuitive understanding of transformers, depicting them as processes that model the trajectory of word particles along the hyper-sphere.

翻译：Transformer模型在自然语言处理领域取得了显著进展，但其内部机制的理解仍具挑战。本文提出一种新颖的几何视角，用以阐明Transformer运算的内在机理。我们的主要贡献在于阐明层归一化如何将隐态特征约束于超球面，进而使注意力机制得以在此曲面上塑造词语的语义表征。这一几何观点将迭代精炼与上下文嵌入等已知特性有机串联。我们通过对预训练1.24亿参数GPT-2模型的探针分析验证了上述见解，发现浅层存在清晰的查询-键注意力模式，并基于先前关于深层注意力头主题特异性的观察进行拓展。借助这些几何洞见，我们提出对Transformer的直观理解，将其描绘为建模词粒子沿超球面运动轨迹的过程。