Recent models have achieved human level performance on the Stanford Question Answering Dataset when using F1 scores to evaluate the reading comprehension task. Yet, teaching machines to comprehend text has not been solved in the general case. By appending one adversarial sentence to the context paragraph, past research has shown that the F1 scores from reading comprehension models drop almost in half. In this paper, I replicate past adversarial research with a new model, ELECTRA-Small, and demonstrate that the new model's F1 score drops from 83.9% to 29.2%. To improve ELECTRA-Small's resistance to this attack, I finetune the model on SQuAD v1.1 training examples with one to five adversarial sentences appended to the context paragraph. Like past research, I find that the finetuned model on one adversarial sentence does not generalize well across evaluation datasets. However, when finetuned on four or five adversarial sentences the model attains an F1 score of more than 70% on most evaluation datasets with multiple appended and prepended adversarial sentences. The results suggest that with enough examples we can make models robust to adversarial attacks.

翻译：最近的研究表明，在使用F1分数评估阅读理解任务时，模型在斯坦福问答数据集上已达到人类水平的表现。然而，在一般情况下，教会机器理解文本尚未得到解决。过去的研究显示，在上下文中附加一个对抗句子会使阅读理解模型的F1分数下降近一半。本文中，我使用新模型ELECTRA-Small复制了过去的对抗研究，并证明该新模型的F1分数从83.9%下降至29.2%。为提高ELECTRA-Small对此攻击的抵抗能力，我基于SQuAD v1.1训练样本（在上下文段落中附加1到5个对抗句子）对该模型进行微调。与过去研究类似，我发现仅用一个对抗句子微调的模型在不同评估数据集上的泛化能力不佳。然而，当使用四个或五个对抗句子进行微调时，模型在大多数评估数据集上（面对多个附加或前置的对抗句子）的F1分数超过70%。结果表明，通过足够多的样本，我们可以使模型对对抗攻击具有稳健性。