This work proposes a novel methodology for measuring compositional behavior in contemporary language embedding models. Specifically, we focus on adjectival modifier phenomena in adjective-noun phrases. In recent years, distributional language representation models have demonstrated great practical success. At the same time, the need for interpretability has elicited questions on their intrinsic properties and capabilities. Crucially, distributional models are often inconsistent when dealing with compositional phenomena in natural language, which has significant implications for their safety and fairness. Despite this, most current research on compositionality is directed towards improving their performance on similarity tasks only. This work takes a different approach, introducing three novel tests of compositional behavior inspired by Montague semantics. Our experimental results indicate that current neural language models do not behave according to the expected linguistic theories. This indicates that current language models may lack the capability to capture the semantic properties we evaluated on limited context, or that linguistic theories from Montagovian tradition may not match the expected capabilities of distributional models.

翻译：本研究提出了一种新颖的方法论，用于衡量当代语言嵌入模型中的组合行为。具体而言，我们聚焦于形容词-名词短语中的形容词修饰现象。近年来，分布式语言表征模型已展现出巨大的实际成功。与此同时，对可解释性的需求引发了关于其内在属性与能力的疑问。关键在于，分布式模型在处理自然语言的组合现象时常常表现出不一致性，这对模型的安全性与公平性具有重要影响。尽管如此，当前大多数关于组合性的研究仅致力于提升模型在相似性任务上的性能。本研究采用了一种不同的路径，引入了三种受蒙塔古语义学启发的新型组合行为测试。我们的实验结果表明，当前的神经语言模型并未按照预期的语言学理论运作。这表明当前的语言模型可能缺乏在有限上下文中捕捉我们所评估的语义属性的能力，或者源自蒙塔古传统的语言学理论可能与分布式模型的预期能力不相匹配。