Pretrained language models have achieved super-human performances on many Machine Reading Comprehension (MRC) benchmarks. Nevertheless, their relative inability to defend against adversarial attacks has spurred skepticism about their natural language understanding. In this paper, we ask whether training with unanswerable questions in SQuAD 2.0 can help improve the robustness of MRC models against adversarial attacks. To explore that question, we fine-tune three state-of-the-art language models on either SQuAD 1.1 or SQuAD 2.0 and then evaluate their robustness under adversarial attacks. Our experiments reveal that current models fine-tuned on SQuAD 2.0 do not initially appear to be any more robust than ones fine-tuned on SQuAD 1.1, yet they reveal a measure of hidden robustness that can be leveraged to realize actual performance gains. Furthermore, we find that the robustness of models fine-tuned on SQuAD 2.0 extends to additional out-of-domain datasets. Finally, we introduce a new adversarial attack to reveal artifacts of SQuAD 2.0 that current MRC models are learning.

翻译：预训练语言模型已在许多机器阅读理解（MRC）基准测试中实现了超越人类的表现。然而，它们在抵御对抗攻击方面相对能力的不足引发了对自然语言理解能力的质疑。本文探究SQuAD 2.0中的不可回答问题训练是否有助于提升MRC模型对抗攻击的鲁棒性。为解答该问题，我们在SQuAD 1.1或SQuAD 2.0上微调三种最先进的语言模型，并评估其在对抗攻击下的鲁棒性。实验表明，当前在SQuAD 2.0上微调的模型初始阶段并不比SQuAD 1.1微调模型表现出更强的鲁棒性，但其蕴藏的可度量的隐藏鲁棒性可通过利用实现实际性能提升。此外，我们发现经SQuAD 2.0微调的模型鲁棒性可扩展到额外的域外数据集。最后，我们引入一种新型对抗攻击，以揭示当前MRC模型正在学习的SQuAD 2.0数据伪影。