Contrastive vision-language representation learning has achieved state-of-the-art performance for zero-shot classification, by learning from millions of image-caption pairs crawled from the internet. However, the massive data that powers large multimodal models such as CLIP, makes them extremely vulnerable to various types of targeted data poisoning and backdoor attacks. Despite this vulnerability, robust contrastive vision-language pre-training against such attacks has remained unaddressed. In this work, we propose ROCLIP, the first effective method for robust pre-training multimodal vision-language models against targeted data poisoning and backdoor attacks. ROCLIP effectively breaks the association between poisoned image-caption pairs by considering a relatively large and varying pool of random captions, and matching every image with the text that is most similar to it in the pool instead of its own caption, every few epochs.It also leverages image and text augmentations to further strengthen the defense and improve the performance of the model. Our extensive experiments show that ROCLIP renders state-of-the-art targeted data poisoning and backdoor attacks ineffective during pre-training CLIP models. In particular, ROCLIP decreases the success rate for targeted data poisoning attacks from 93.75% to 12.5% and that of backdoor attacks down to 0%, while improving the model's linear probe performance by 10% and maintains a similar zero shot performance compared to CLIP. By increasing the frequency of matching, ROCLIP is able to defend strong attacks, which add up to 1% poisoned examples to the data, and successfully maintain a low attack success rate of 12.5%, while trading off the performance on some tasks.

翻译：对比式视觉-语言表征学习通过从互联网抓取的数百万图像-文本对中学习，在零样本分类任务上取得了最先进性能。然而，支撑CLIP等大型多模态模型的庞大数据集使其极易遭受各类定向数据投毒和后门攻击。尽管存在这一脆弱性，针对此类攻击的鲁棒对比视觉-语言预训练至今仍未被解决。本文提出ROCLIP，这是首个针对定向数据投毒和后门攻击实现多模态视觉-语言模型鲁棒预训练的有效方法。ROCLIP通过建立包含随机文本的较大且动态变化的池，每隔若干训练周期将每张图像与池中最相似的文本（而非其原始文本）进行匹配，有效破坏被投毒图像-文本对的关联。同时利用图像和文本增强技术进一步强化防御并提升模型性能。大量实验表明，ROCLIP能在预训练CLIP模型时使最先进的定向数据投毒和后门攻击失效。具体而言，ROCLIP将定向数据投毒攻击成功率从93.75%降至12.5%，后门攻击成功率降至0%，同时将模型的线性探测性能提升10%，并保持与CLIP相当的零样本性能。通过提高匹配频率，ROCLIP能防御包含高达1%投毒样本的强攻击，成功将攻击成功率维持在12.5%的低水平，仅在部分任务上存在性能权衡。