Vision-language pre-training models (VLPs) have exhibited revolutionary improvements in various vision-language tasks. In VLP, some adversarial attacks fool a model into false or absurd classifications. Previous studies addressed these attacks by fine-tuning the model or changing its architecture. However, these methods risk losing the original model's performance and are difficult to apply to downstream tasks. In particular, their applicability to other tasks has not been considered. In this study, we addressed the reduction of the impact of typographic attacks on CLIP without changing the model parameters. To achieve this, we expand the idea of ``prefix learning'' and introduce our simple yet effective method: Defense-Prefix (DP), which inserts the DP token before a class name to make words ``robust'' against typographic attacks. Our method can be easily applied to downstream tasks, such as object detection, because the proposed method is independent of the model parameters. Our method significantly improves the accuracy of classification tasks for typographic attack datasets, while maintaining the zero-shot capabilities of the model. In addition, we leverage our proposed method for object detection, demonstrating its high applicability and effectiveness. The codes and datasets will be publicly available.

翻译：视觉-语言预训练模型（VLP）在各类视觉-语言任务中展现出革命性改进。在VLP中，某些对抗性攻击会诱使模型做出错误或荒谬的分类。以往研究通过微调模型或变更其架构来应对这些攻击。然而，这些方法存在丢失原始模型性能的风险，且难以应用于下游任务。特别地，它们对其他任务的适用性尚未得到充分考虑。在本研究中，我们致力于在不改变模型参数的前提下，降低文字攻击对CLIP模型的影响。为此，我们拓展了"前缀学习"的概念，提出一种简洁而有效的方法——防御前缀（Defense-Prefix, DP），该方法通过在类别名称前插入DP标记，使词汇对文字攻击具有"鲁棒性"。由于本方法独立于模型参数，可轻松应用于目标检测等下游任务。我们的方法在显著提升文字攻击数据集分类任务准确率的同时，保持了模型的零样本能力。此外，我们将所提方法拓展至目标检测任务，证明了其高度的适用性与有效性。相关代码与数据集将公开提供。