The success of pre-trained contextualized representations has prompted researchers to analyze them for the presence of linguistic information. Indeed, it is natural to assume that these pre-trained representations do encode some level of linguistic knowledge as they have brought about large empirical improvements on a wide variety of NLP tasks, which suggests they are learning true linguistic generalization. In this work, we focus on intrinsic probing, an analysis technique where the goal is not only to identify whether a representation encodes a linguistic attribute but also to pinpoint where this attribute is encoded. We propose a novel latent-variable formulation for constructing intrinsic probes and derive a tractable variational approximation to the log-likelihood. Our results show that our model is versatile and yields tighter mutual information estimates than two intrinsic probes previously proposed in the literature. Finally, we find empirical evidence that pre-trained representations develop a cross-lingually entangled notion of morphosyntax.

翻译：预训练上下文表示的成功促使研究者分析其中是否包含语言信息。实际上，人们自然假定这些预训练表示确实编码了某种程度的语言知识，因为它们在众多自然语言处理任务上带来了显著的实证改进，这表明它们正在学习真正的语言泛化。在本工作中，我们聚焦于内在探测这一分析技术，其目标不仅在于识别表示是否编码了语言属性，更在于定位该属性编码的位置。我们提出了一种新颖的隐变量公式用于构建内在探测器，并推导出对数似然的易处理变分近似。结果表明，我们的模型具有通用性，且比文献中先前提出的两种内在探测器能获得更紧密的互信息估计。最后，我们发现预训练表示发展出了一种跨语言纠缠的形态句法概念的实证证据。