Recent studies of the emergent capabilities of transformer-based Natural Language Understanding (NLU) models have indicated that they have an understanding of lexical and compositional semantics. We provide evidence that suggests these claims should be taken with a grain of salt: we find that state-of-the-art Natural Language Inference (NLI) models are sensitive towards minor semantics preserving surface-form variations, which lead to sizable inconsistent model decisions during inference. Notably, this behaviour differs from valid and in-depth comprehension of compositional semantics, however does neither emerge when evaluating model accuracy on standard benchmarks nor when probing for syntactic, monotonic, and logically robust reasoning. We propose a novel framework to measure the extent of semantic sensitivity. To this end, we evaluate NLI models on adversarially generated examples containing minor semantics-preserving surface-form input noise. This is achieved using conditional text generation, with the explicit condition that the NLI model predicts the relationship between the original and adversarial inputs as a symmetric equivalence entailment. We systematically study the effects of the phenomenon across NLI models for \emph{in-} and \emph{out-of} domain settings. Our experiments show that semantic sensitivity causes performance degradations of $12.92\%$ and $23.71\%$ average over \emph{in-} and \emph{out-of-} domain settings, respectively. We further perform ablation studies, analysing this phenomenon across models, datasets, and variations in inference and show that semantic sensitivity can lead to major inconsistency within model predictions.

翻译：近期关于基于Transformer的自然语言理解（NLU）模型涌现能力的研究表明，这些模型能够理解词汇语义和组合语义。我们提供的证据表明，这些结论应持审慎态度：我们发现，最先进的自然语言推理（NLI）模型对微小的、保留语义的表面形式变化十分敏感，这种敏感性会导致推理过程中出现显著的不一致决策。值得注意的是，这一行为与对组合语义的有效深入理解有所不同，但既未在标准基准测试的模型准确率评估中显现，也未在对句法、单调性和逻辑鲁棒性的探究中被发现。我们提出了一种新颖的框架来衡量语义敏感性的程度。为此，我们使用对抗性生成的示例（包含微小的、保留语义的表面形式输入噪声）来评估NLI模型。这通过条件文本生成实现，其显式条件为：NLI模型将原始输入与对抗性输入之间的关系预测为对称等价蕴含关系。我们系统研究了该现象在域内和域外设置下对NLI模型的影响。实验表明，语义敏感性导致域内和域外设置的平均性能分别下降12.92%和23.71%。我们进一步进行了消融研究，分析了该现象在不同模型、数据集和推理变体中的表现，并表明语义敏感性可能导致模型预测内出现重大不一致性。