In recent years, distributional language representation models have demonstrated great practical success. At the same time, the need for interpretability has elicited questions on their intrinsic properties and capabilities. Crucially, distributional models are often inconsistent when dealing with compositional phenomena in natural language, which has significant implications for their safety and fairness. Despite this, most current research on compositionality is directed towards improving their performance on similarity tasks only. This work takes a different approach, and proposes a methodology for measuring compositional behavior in contemporary language models. Specifically, we focus on adjectival modifier phenomena in adjective-noun phrases. We introduce three novel tests of compositional behavior inspired by Montague semantics. Our experimental results indicate that current neural language models behave according to the expected linguistic theories to a limited extent only. This raises the question of whether these language models are not able to capture the semantic properties we evaluated, or whether linguistic theories from Montagovian tradition would not match the expected capabilities of distributional models.

翻译：近年来，分布式语言表征模型取得了巨大的实际成功。与此同时，对可解释性的需求引发了对其内在属性与能力的探讨。关键问题在于，分布式模型在处理自然语言中的组合现象时往往不一致，这对模型的安全性和公平性具有重要影响。尽管如此，当前关于组合性的研究大多仅致力于提升模型在相似性任务上的性能。本研究另辟蹊径，提出了一种用于测量当代语言模型中组合行为的方法体系。具体而言，我们聚焦于形容词-名词短语中的形容词修饰现象。受蒙塔古语义学启发，我们引入了三项新颖的组合行为测试。实验结果表明，当前神经语言模型仅在有限程度上符合预期语言学理论。这引出一个问题：究竟是这些语言模型未能捕捉我们所评估的语义属性，抑或蒙塔古传统下的语言学理论与分布式模型的预期能力并不匹配。