Shortcut reasoning is an irrational process of inference, which degrades the robustness of an NLP model. While a number of previous work has tackled the identification of shortcut reasoning, there are still two major limitations: (i) a method for quantifying the severity of the discovered shortcut reasoning is not provided; (ii) certain types of shortcut reasoning may be missed. To address these issues, we propose a novel method for identifying shortcut reasoning. The proposed method quantifies the severity of the shortcut reasoning by leveraging out-of-distribution data and does not make any assumptions about the type of tokens triggering the shortcut reasoning. Our experiments on Natural Language Inference and Sentiment Analysis demonstrate that our framework successfully discovers known and unknown shortcut reasoning in the previous work.

翻译：捷径推理是一种非理性的推理过程，会降低自然语言处理模型的鲁棒性。尽管已有大量研究致力于识别捷径推理，但仍存在两大局限：（i）缺乏量化已发现捷径推理严重程度的方法；（ii）可能遗漏某些类型的捷径推理。为解决这些问题，我们提出了一种识别捷径推理的新方法。该方法利用分布外数据量化捷径推理的严重程度，且不对触发捷径推理的令牌类型做出任何假设。我们在自然语言推理和情感分析任务上的实验表明，该框架成功发现了已有研究中已知和未知的捷径推理。