The question of what kinds of linguistic information are encoded in different layers of Transformer-based language models is of considerable interest for the NLP community. Existing work, however, has overwhelmingly focused on word-level representations and encoder-only language models with the masked-token training objective. In this paper, we present experiments with semantic structural probing, a method for studying sentence-level representations via finding a subspace of the embedding space that provides suitable task-specific pairwise distances between data-points. We apply our method to language models from different families (encoder-only, decoder-only, encoder-decoder) and of different sizes in the context of two tasks, semantic textual similarity and natural-language inference. We find that model families differ substantially in their performance and layer dynamics, but that the results are largely model-size invariant.

翻译：Transformer系列语言模型各层编码了何种语言信息，这一问题引起了自然语言处理学界的广泛关注。然而，现有研究主要聚焦于词级表示以及采用掩码标记训练目标、仅含编码器的语言模型。本文通过语义结构探针法进行实验，该方法通过寻找嵌入空间中能提供数据点间合适任务特定成对距离的子空间，来研究句子级表示。我们将该方法应用于不同家族（仅编码器、仅解码器、编码器-解码器）及不同规模的语言模型，在语义文本相似度和自然语言推理两项任务中进行评估。研究发现，各模型家族在性能表现与层级动态上存在显著差异，但模型规模对结果的影响基本不显著。