The CLIP (Contrastive Language-Image Pre-training) model and its variants are becoming the de facto backbone in many applications. However, training a CLIP model from hundreds of millions of image-text pairs can be prohibitively expensive. Furthermore, the conventional CLIP model doesn't differentiate between the visual semantics and meaning of text regions embedded in images. This can lead to non-robustness when the text in the embedded region doesn't match the image's visual appearance. In this paper, we discuss two effective approaches to improve the efficiency and robustness of CLIP training: (1) augmenting the training dataset while maintaining the same number of optimization steps, and (2) filtering out samples that contain text regions in the image. By doing so, we significantly improve the classification and retrieval accuracy on public benchmarks like ImageNet and CoCo. Filtering out images with text regions also protects the model from typographic attacks. To verify this, we build a new dataset named ImageNet with Adversarial Text Regions (ImageNet-Attr). Our filter-based CLIP model demonstrates a top-1 accuracy of 68.78\%, outperforming previous models whose accuracy was all below 50\%.

翻译：CLIP（对比语言-图像预训练）模型及其变体正逐渐成为众多应用中的事实标准主干网络。然而，从数亿图文对中训练CLIP模型的计算成本极高。此外，传统CLIP模型无法区分图像中嵌入文本区域的视觉语义与文本含义，当嵌入区域的文本与图像视觉外观不匹配时，可能导致模型鲁棒性不足。本文探讨了两种提升CLIP训练效率与鲁棒性的有效方法：（1）在保持优化步数不变的情况下增强训练数据集；（2）过滤包含图像文本区域的样本。通过上述方法，我们在ImageNet和CoCo等公开基准测试中显著提升了分类与检索精度。过滤含文本区域的图像还能使模型免受文字攻击。为验证这一点，我们构建了名为ImageNet-Attr（含对抗文本区域的ImageNet）的新数据集。基于过滤机制的CLIP模型实现了68.78%的Top-1准确率，优于此前所有准确率低于50%的模型。