The self-attention mechanism has significantly advanced the field of natural language processing, facilitating the development of advanced language-learning machines. Although its utility is widely acknowledged, the precise mechanisms of self-attention underlying its advanced learning and the quantitative characterization of this learning process remains an open research question. This study introduces a new approach for quantifying information processing within the self-attention mechanism. The analysis conducted on the BERT-12 architecture reveals that, in the final layers, the attention map focuses on sentence separator tokens, suggesting a practical approach to text segmentation based on semantic features. Based on the vector space emerging from the self-attention heads, a context similarity matrix, measuring the scalar product between two token vectors was derived, revealing distinct similarities between different token vector pairs within each head and layer. The findings demonstrated that different attention heads within an attention block focused on different linguistic characteristics, such as identifying token repetitions in a given text or recognizing a token of common appearance in the text and its surrounding context. This specialization is also reflected in the distribution of distances between token vectors with high similarity as the architecture progresses. The initial attention layers exhibit substantially long-range similarities; however, as the layers progress, a more short-range similarity develops, culminating in a preference for attention heads to create strong similarities within the same sentence. Finally, the behavior of individual heads was analyzed by examining the uniqueness of their most common tokens in their high similarity elements. Each head tends to focus on a unique token from the text and builds similarity pairs centered around it.

翻译：自注意力机制显著推动了自然语言处理领域的发展，促进了高级语言学习机器的开发。尽管其效用已得到广泛认可，但支撑其高级学习过程的自注意力精确机制以及该学习过程的量化表征仍是一个开放的研究问题。本研究提出了一种量化自注意力机制内部信息处理的新方法。对BERT-12架构的分析表明，在最终层中，注意力图聚焦于句子分隔符标记，这提示了一种基于语义特征的文本分割实用方法。基于自注意力头产生的向量空间，我们推导了一个上下文相似性矩阵，该矩阵通过测量两个标记向量之间的标量积来量化其关系，揭示了每个注意力头和层内不同标记向量对之间的显著相似性。研究结果表明，注意力块内的不同注意力头专注于不同的语言特征，例如识别给定文本中的标记重复，或识别文本中常见出现的标记及其周围上下文。这种专业化也反映在随着架构推进，高相似性标记向量之间距离的分布变化上。初始注意力层表现出显著的长距离相似性；然而，随着层数增加，更短距离的相似性逐渐形成，最终注意力头倾向于在同一句子内建立强相似性。最后，通过检查各注意力头在其高相似性元素中最常见标记的唯一性，我们分析了单个注意力头的行为。每个注意力头倾向于聚焦于文本中的一个独特标记，并围绕该标记构建相似性对。