Vision-Language models like CLIP have been shown to be highly effective at linking visual perception and natural language understanding, enabling sophisticated image-text capabilities, including strong retrieval and zero-shot classification performance. Their widespread use, as well as the fact that CLIP models are trained on image-text pairs from the web, make them both a worthwhile and relatively easy target for backdoor attacks. As training foundational models, such as CLIP, from scratch is very expensive, this paper focuses on cleaning potentially poisoned models via fine-tuning. We first show that existing cleaning techniques are not effective against simple structured triggers used in Blended or BadNet backdoor attacks, exposing a critical vulnerability for potential real-world deployment of these models. Then, we introduce PAR, Perturb and Recover, a surprisingly simple yet effective mechanism to remove backdoors from CLIP models. Through extensive experiments across different encoders and types of backdoor attacks, we show that PAR achieves high backdoor removal rate while preserving good standard performance. Finally, we illustrate that our approach is effective even only with synthetic text-image pairs, i.e. without access to real training data. The code and models are available on \href{https://github.com/nmndeep/PerturbAndRecover}{GitHub}.

翻译：视觉-语言模型（如CLIP）已被证明在链接视觉感知与自然语言理解方面非常有效，能够实现复杂的图像-文本能力，包括强大的检索和零样本分类性能。这些模型的广泛使用，以及CLIP模型是在网络图像-文本对上进行训练的事实，使其成为后门攻击的有价值且相对容易的目标。由于从头训练基础模型（如CLIP）非常昂贵，本文专注于通过微调来清理可能被污染的模型。我们首先证明，现有清理技术对Blended或BadNet后门攻击中使用的简单结构化触发器无效，暴露了这些模型在现实世界部署中的关键漏洞。随后，我们提出PAR（扰动与恢复），一种令人惊讶地简单但有效的机制，用于移除CLIP模型中的后门。通过跨不同编码器和后门攻击类型的广泛实验，我们表明PAR在保持良好标准性能的同时实现了高后门移除率。最后，我们证明即便仅使用合成图像-文本对（即无法访问真实训练数据），我们的方法仍然有效。代码和模型可在GitHub上获取：\href{https://github.com/nmndeep/PerturbAndRecover}{GitHub}。