Understanding the inner workings of neural network models is a crucial step for rationalizing their output and refining their architecture. Transformer-based models are the core of recent natural language processing and have been analyzed typically with attention patterns as their epoch-making feature is contextualizing surrounding input words via attention mechanisms. In this study, we analyze their inner contextualization by considering all the components, including the feed-forward block (i.e., a feed-forward layer and its surrounding residual and normalization layers) as well as the attention. Our experiments with masked language models show that each of the previously overlooked components did modify the degree of the contextualization in case of processing special word-word pairs (e.g., consisting of named entities). Furthermore, we find that some components cancel each other's effects. Our results could update the typical view about each component's roles (e.g., attention performs contextualization, and the other components serve different roles) in the Transformer layer.

翻译：理解神经网络模型的内部运作机制是合理化其输出并优化其架构的关键步骤。基于Transformer的模型是当前自然语言处理的核心，通常通过注意力模式进行分析，因为其划时代特征是通过注意力机制将周围输入词汇进行上下文化。在本研究中，我们通过考虑所有组件（包括前馈模块，即前馈层及其周围的残差层和归一化层）以及注意力机制，分析了模型的内部上下文化过程。针对掩码语言模型的实验表明，此前被忽视的每个组件在处理特殊词对（例如由命名实体构成的词对）时，都会对上下文化程度产生影响。此外，我们发现某些组件会相互抵消效应。这一结果可能更新人们对Transformer层中各组件角色的传统认知（例如注意力执行上下文化，而其他组件承担不同功能）。