Pretrained language models are expected to effectively map input text to a set of vectors while preserving the inherent relationships within the text. Consequently, designing a white-box model to compute metrics that reflect the presence of specific internal relations in these vectors has become a common approach for post-hoc interpretability analysis of pretrained language models. However, achieving interpretability in white-box models and ensuring the rigor of metric computation becomes challenging when the source model lacks inherent interpretability. Therefore, in this paper, we discuss striking a balance in this trade-off and propose a novel line to constructing metrics for understanding the mechanisms of pretrained language models. We have specifically designed a family of metrics along this line of investigation, and the model used to compute these metrics is referred to as the tree topological probe. We conducted measurements on BERT-large by using these metrics. Based on the experimental results, we propose a speculation regarding the working mechanism of BERT-like pretrained language models, as well as a strategy for enhancing fine-tuning performance by leveraging the topological probe to improve specific submodules.

翻译：预训练语言模型期望能有效将输入文本映射到一组向量，同时保留文本内部固有的关系。因此，设计一个白盒模型来计算反映这些向量中特定内部关系存在性的度量，已成为对预训练语言模型进行事后可解释性分析的常见方法。然而，当源模型本身缺乏固有可解释性时，实现白盒模型的可解释性并确保度量计算的严谨性就变得具有挑战性。因此，在本文中，我们探讨了在这两者之间寻求平衡，并提出了一条构建用于理解预训练语言模型机制的度量的新思路。我们沿着这一研究思路专门设计了一组度量，用于计算这些度量的模型被称为树拓扑探测模型。我们利用这些度量对BERT-large进行了测量。基于实验结果，我们提出了一个关于类似BERT的预训练语言模型工作机制的推测，以及一种通过利用拓扑探测模型改进特定子模块来增强微调性能的策略。