This study investigates the ability of perceptron-type neurons in language models to perform intra-neuronal attention; that is, to identify different homogeneous categorical segments within the synthetic thought category they encode, based on a segmentation of specific activation zones for the tokens to which they are particularly responsive. The objective of this work is therefore to determine to what extent formal neurons can establish a homomorphic relationship between activation-based and categorical segmentations. The results suggest the existence of such a relationship, albeit tenuous, only at the level of tokens with very high activation levels. This intra-neuronal attention subsequently enables categorical restructuring processes at the level of neurons in the following layer, thereby contributing to the progressive formation of high-level categorical abstractions.

翻译：本研究探讨了语言模型中感知器型神经元执行神经元内注意力的能力；即基于对它们特别响应的标记的特定激活区域进行分割，识别其编码的合成思维类别内不同的同质类别片段。因此，本工作的目标是确定形式神经元能在何种程度上建立基于激活的分割与类别分割之间的同态关系。结果表明，仅在具有极高激活水平的标记层面存在这种关系，尽管这种关系较为微弱。这种神经元内注意力随后能够在下一层神经元层面实现类别重组过程，从而促进高层次类别抽象的渐进形成。