Vision-language pre-training models (VLPs) have exhibited revolutionary improvements in various vision-language tasks. In VLP, some adversarial attacks fool a model into false or absurd classifications. Previous studies addressed these attacks by fine-tuning the model or changing its architecture. However, these methods risk losing the original model's performance and are difficult to apply to downstream tasks. In particular, their applicability to other tasks has not been considered. In this study, we addressed the reduction of the impact of typographic attacks on CLIP without changing the model parameters. To achieve this, we expand the idea of "prefix learning" and introduce our simple yet effective method: Defense-Prefix (DP), which inserts the DP token before a class name to make words "robust" against typographic attacks. Our method can be easily applied to downstream tasks, such as object detection, because the proposed method is independent of the model parameters. Our method significantly improves the accuracy of classification tasks for typographic attack datasets, while maintaining the zero-shot capabilities of the model. In addition, we leverage our proposed method for object detection, demonstrating its high applicability and effectiveness. The codes and datasets are available at https://github.com/azuma164/Defense-Prefix.

翻译：视觉语言预训练模型（VLPs）在多种视觉语言任务中展现出革命性进步。在VLP中，某些对抗性攻击会诱导模型做出错误或荒谬的分类。以往研究通过微调模型或改变架构来应对这些攻击，但此类方法存在丢失原始模型性能的风险，且难以应用于下游任务——尤其是其跨任务适用性尚未被充分考量。本研究聚焦于在不改变模型参数的前提下，降低排版攻击对CLIP模型的影响。为实现该目标，我们扩展了“前缀学习”概念，提出简洁高效的“防御前缀”（DP）方法：通过在类别名称前插入DP标记，使词语具备抵御排版攻击的鲁棒性。由于该方法独立于模型参数，可轻松适配目标检测等下游任务。在排版攻击数据集上，我们的方法在保持模型零样本能力的同时，显著提升了分类任务的准确率。此外，我们将该方法拓展应用于目标检测，验证了其卓越的适用性与有效性。相关代码与数据集已开源至 https://github.com/azuma164/Defense-Prefix。