Many adversarial attacks in NLP perturb inputs to produce visually similar strings ('ergo' $\rightarrow$ '$\epsilon$rgo') which are legible to humans but degrade model performance. Although preserving legibility is a necessary condition for text perturbation, little work has been done to systematically characterize it; instead, legibility is typically loosely enforced via intuitions around the nature and extent of perturbations. Particularly, it is unclear to what extent can inputs be perturbed while preserving legibility, or how to quantify the legibility of a perturbed string. In this work, we address this gap by learning models that predict the legibility of a perturbed string, and rank candidate perturbations based on their legibility. To do so, we collect and release \dataset, a human-annotated dataset comprising the legibility of visually perturbed text. Using this dataset, we build both text- and vision-based models which achieve up to $0.91$ F1 score in predicting whether an input is legible, and an accuracy of $0.86$ in predicting which of two given perturbations is more legible. Additionally, we discover that legible perturbations from the \dataset dataset are more effective at lowering the performance of NLP models than best-known attack strategies, suggesting that current models may be vulnerable to a broad range of perturbations beyond what is captured by existing visual attacks. Data, code, and models are available at https://github.com/dvsth/learning-legibility-2023.

翻译：许多自然语言处理中的对抗性攻击通过扰动输入产生视觉上相似的字符串（例如'ergo' → 'εrgo'），这些字符串对人类可读但会降低模型性能。尽管保持可读性是文本扰动的必要条件，但很少有工作系统性地刻画这一特性；相反，可读性通常通过关于扰动性质和范围的直觉进行粗略约束。尤其，目前尚不清楚输入能在多大程度上被扰动同时保持可读性，以及如何量化扰动字符串的可读性。在本工作中，我们通过训练预测扰动字符串可读性的模型，并基于可读性对候选扰动进行排序，填补了这一空白。为此，我们收集并发布了\dataset数据集，这是一个包含视觉扰动文本可读性的人工标注数据集。利用该数据集，我们构建了基于文本和基于视觉的模型，在预测输入是否可读方面取得了最高0.91的F1分数，在预测两个给定扰动中哪一个更可读方面取得了0.86的准确率。此外，我们发现\dataset数据集中的可读扰动在降低自然语言处理模型性能方面比已知的最佳攻击策略更有效，这表明当前模型可能容易受到现有视觉攻击所涵盖范围之外的广泛扰动的影响。数据、代码和模型可在https://github.com/dvsth/learning-legibility-2023获取。