Recent studies of the emergent capabilities of transformer-based Natural Language Understanding (NLU) models have indicated that they have an understanding of lexical and compositional semantics. We provide evidence that suggests these claims should be taken with a grain of salt: we find that state-of-the-art Natural Language Inference (NLI) models are sensitive towards minor semantics preserving surface-form variations, which lead to sizable inconsistent model decisions during inference. Notably, this behaviour differs from valid and in-depth comprehension of compositional semantics, however does neither emerge when evaluating model accuracy on standard benchmarks nor when probing for syntactic, monotonic, and logically robust reasoning. We propose a novel framework to measure the extent of semantic sensitivity. To this end, we evaluate NLI models on adversarially generated examples containing minor semantics-preserving surface-form input noise. This is achieved using conditional text generation, with the explicit condition that the NLI model predicts the relationship between the original and adversarial inputs as a symmetric equivalence entailment. We systematically study the effects of the phenomenon across NLI models for $\textbf{in-}$ and $\textbf{out-of-}$ domain settings. Our experiments show that semantic sensitivity causes performance degradations of $12.92\%$ and $23.71\%$ average over $\textbf{in-}$ and $\textbf{out-of-}$ domain settings, respectively. We further perform ablation studies, analysing this phenomenon across models, datasets, and variations in inference and show that semantic sensitivity can lead to major inconsistency within model predictions.

翻译：近期关于基于Transformer的自然语言理解（NLU）模型涌现能力的研究表明，这些模型已具备词汇语义和组合语义的理解能力。我们提供的证据表明，应谨慎看待这些论断：研究发现，最先进的自然语言推理（NLI）模型对微小的语义保持表层形式变化具有敏感性，这种敏感性会在推理过程中导致显著的不一致模型决策。值得注意的是，这种行为与对组合语义的有效深度理解存在差异，并且在标准基准测试的模型准确率评估中，或针对句法、单调性及逻辑鲁棒推理的探测中均未显现。我们提出了一种新颖的框架来度量语义敏感性的程度。为此，我们在包含微小语义保持表层形式输入噪声的对抗生成样本上评估NLI模型。这通过条件文本生成实现，其明确条件为：NLI模型将原始输入与对抗输入之间的关系预测为对称等价蕴含。我们系统研究了该现象在$\textbf{域内}$和$\textbf{域外}$设置下对不同NLI模型的影响。实验表明，语义敏感性在$\textbf{域内}$和$\textbf{域外}$设置下分别导致平均$12.92\%$和$23.71\%$的性能下降。我们进一步进行了消融研究，分析了该现象在不同模型、数据集和推理变体中的表现，并证明语义敏感性可能导致模型预测中的显著不一致性。