Transformer-based models have made remarkable advancements in various NLP areas. Nevertheless, these models often exhibit vulnerabilities when confronted with adversarial attacks. In this paper, we explore the effect of quantization on the robustness of Transformer-based models. Quantization usually involves mapping a high-precision real number to a lower-precision value, aiming at reducing the size of the model at hand. To the best of our knowledge, this work is the first application of quantization on the robustness of NLP models. In our experiments, we evaluate the impact of quantization on BERT and DistilBERT models in text classification using SST-2, Emotion, and MR datasets. We also evaluate the performance of these models against TextFooler, PWWS, and PSO adversarial attacks. Our findings show that quantization significantly improves (by an average of 18.68%) the adversarial accuracy of the models. Furthermore, we compare the effect of quantization versus that of the adversarial training approach on robustness. Our experiments indicate that quantization increases the robustness of the model by 18.80% on average compared to adversarial training without imposing any extra computational overhead during training. Therefore, our results highlight the effectiveness of quantization in improving the robustness of NLP models.

翻译：基于Transformer的模型在各种自然语言处理领域取得了显著进展。然而，这些模型在面对对抗攻击时往往表现出脆弱性。在本文中，我们探讨了量化对基于Transformer模型鲁棒性的影响。量化通常涉及将高精度实数映射为低精度值，旨在减小当前模型的规模。据我们所知，本研究是首次将量化应用于提升NLP模型鲁棒性的工作。在我们的实验中，我们使用SST-2、Emotion和MR数据集评估了量化对BERT和DistilBERT模型在文本分类中的影响。同时，我们还评估了这些模型对抗TextFooler、PWWS和PSO攻击的性能表现。研究结果表明，量化能够显著提升模型的对抗性准确率（平均提升18.68%）。此外，我们比较了量化方法与对抗训练方法对鲁棒性的影响。实验表明，与对抗训练相比，量化在训练过程中不增加额外计算开销的情况下，平均使模型鲁棒性提升18.80%。因此，我们的结果突出了量化在提升NLP模型鲁棒性方面的有效性。