A recent line of work in NLP focuses on the (dis)ability of models to generalise compositionally for artificial languages. However, when considering natural language tasks, the data involved is not strictly, or locally, compositional. Quantifying the compositionality of data is a challenging task, which has been investigated primarily for short utterances. We use recursive neural models (Tree-LSTMs) with bottlenecks that limit the transfer of information between nodes. We illustrate that comparing data's representations in models with and without the bottleneck can be used to produce a compositionality metric. The procedure is applied to the evaluation of arithmetic expressions using synthetic data, and sentiment classification using natural language data. We demonstrate that compression through a bottleneck impacts non-compositional examples disproportionately and then use the bottleneck compositionality metric (BCM) to distinguish compositional from non-compositional samples, yielding a compositionality ranking over a dataset.

翻译：自然语言处理领域近期的研究方向聚焦于模型对人工语言组合泛化的（不）能力。然而，在处理自然语言任务时，所涉及的数据并非严格或局部地具有组合性。量化数据的组合性是一项具有挑战性的任务，此前主要针对短语句进行研究。我们采用带有瓶颈的递归神经模型（Tree-LSTMs），通过限制节点间的信息传递，证明比较有无瓶颈模型中数据的表征可用于构建组合性度量指标。该方法被应用于算术表达式评估（基于合成数据）和情感分类（基于自然语言数据）中。我们验证了瓶颈压缩对非组合性样本的影响显著更大，进而利用瓶颈组合性度量（BCM）区分组合性与非组合性样本，最终对数据集形成组合性排序。