In recent years, NLP practitioners have converged on the following practice: (i) import an off-the-shelf pretrained (masked) language model; (ii) append a multilayer perceptron atop the CLS token's hidden representation (with randomly initialized weights); and (iii) fine-tune the entire model on a downstream task (MLP). This procedure has produced massive gains on standard NLP benchmarks, but these models remain brittle, even to mild adversarial perturbations, such as word-level synonym substitutions. In this work, we demonstrate surprising gains in adversarial robustness enjoyed by Model-tuning Via Prompts (MVP), an alternative method of adapting to downstream tasks. Rather than modifying the model (by appending an MLP head), MVP instead modifies the input (by appending a prompt template). Across three classification datasets, MVP improves performance against adversarial word-level synonym substitutions by an average of 8% over standard methods and even outperforms adversarial training-based state-of-art defenses by 3.5%. By combining MVP with adversarial training, we achieve further improvements in robust accuracy while maintaining clean accuracy. Finally, we conduct ablations to investigate the mechanism underlying these gains. Notably, we find that the main causes of vulnerability of MLP can be attributed to the misalignment between pre-training and fine-tuning tasks, and the randomly initialized MLP parameters. Code is available at https://github.com/acmi-lab/mvp

翻译：近年来，NLP从业者逐渐采纳以下实践：（i）导入预训练的（掩码）语言模型；（ii）在CLS令牌的隐藏表示之上附加一个多层感知器（权重随机初始化）；以及（iii）在下游任务上微调整个模型（MLP）。这一流程在标准NLP基准测试中取得了巨大提升，但这些模型仍然脆弱，甚至难以应对微弱的对抗扰动，例如词级同义词替换。在本工作中，我们展示了基于提示的模型调优（MVP）在对抗鲁棒性方面的显著提升，这是一种替代性的下游任务适配方法。MVP不修改模型（通过附加MLP头），而是修改输入（通过附加提示模板）。在三个分类数据集上，MVP对抗词级同义词替换的性能平均比标准方法高出8%，甚至比基于对抗训练的最先进防御方法高出3.5%。通过将MVP与对抗训练结合，我们在保持干净精度的同时进一步提升了鲁棒精度。最后，我们进行消融实验以探究这些增益背后的机制。值得注意的是，我们发现MLP脆弱性的主要原因可归因于预训练与微调任务之间的不匹配以及随机初始化的MLP参数。代码见https://github.com/acmi-lab/mvp