Compared to English, German word order is freer and therefore poses additional challenges for natural language inference (NLI). We create WOGLI (Word Order in German Language Inference), the first adversarial NLI dataset for German word order that has the following properties: (i) each premise has an entailed and a non-entailed hypothesis; (ii) premise and hypotheses differ only in word order and necessary morphological changes to mark case and number. In particular, each premise andits two hypotheses contain exactly the same lemmata. Our adversarial examples require the model to use morphological markers in order to recognise or reject entailment. We show that current German autoencoding models fine-tuned on translated NLI data can struggle on this challenge set, reflecting the fact that translated NLI datasets will not mirror all necessary language phenomena in the target language. We also examine performance after data augmentation as well as on related word order phenomena derived from WOGLI. Our datasets are publically available at https://github.com/ireinig/wogli.

翻译：相比英语，德语词序更为自由，这为自然语言推理(NLI)带来了额外挑战。我们构建了首个针对德语词序的对抗性NLI数据集WOGLI（德语言语推理中的词序），其具有以下特性：(i) 每个前提均包含蕴含与非蕴含两种假设；(ii) 前提与假设仅在词序及必要的形态变化（标记格和数）上存在差异。特别地，每个前提及其两种假设包含完全相同的词元。我们的对抗性样本要求模型利用形态标记来识别或拒绝蕴含关系。实验表明，当前基于翻译NLI数据微调的德语自编码模型在此挑战集上存在困难，这反映出翻译NLI数据集无法完整映射目标语言中所有必要的语言现象。我们还研究了数据增强后的性能，以及源自WOGLI的相关词序现象。数据集已在https://github.com/ireinig/wogli 公开。