A key benefit of deep vision-language models such as CLIP is that they enable zero-shot open vocabulary classification; the user has the ability to define novel class labels via natural language prompts at inference time. However, while CLIP-based zero-shot classifiers have demonstrated competitive performance across a range of domain shifts, they remain highly vulnerable to adversarial attacks. Therefore, ensuring the robustness of such models is crucial for their reliable deployment in the wild. In this work, we introduce Open Vocabulary Certification (OVC), a fast certification method designed for open-vocabulary models like CLIP via randomized smoothing techniques. Given a base "training" set of prompts and their corresponding certified CLIP classifiers, OVC relies on the observation that a classifier with a novel prompt can be viewed as a perturbed version of nearby classifiers in the base training set. Therefore, OVC can rapidly certify the novel classifier using a variation of incremental randomized smoothing. By using a caching trick, we achieve approximately two orders of magnitude acceleration in the certification process for novel prompts. To achieve further (heuristic) speedups, OVC approximates the embedding space at a given input using a multivariate normal distribution bypassing the need for sampling via forward passes through the vision backbone. We demonstrate the effectiveness of OVC on through experimental evaluation using multiple vision-language backbones on the CIFAR-10 and ImageNet test datasets.

翻译：深度视觉语言模型（如CLIP）的一个关键优势在于，它们能够实现零样本开放词汇分类：用户可以在推理时通过自然语言提示定义新颖的类别标签。然而，尽管基于CLIP的零样本分类器在多种领域偏移下展现出具有竞争力的性能，它们仍然极易受到对抗性攻击。因此，确保此类模型的鲁棒性对其在真实场景中的可靠部署至关重要。本文提出开放词汇认证（OVC），一种通过随机平滑技术为CLIP等开放词汇模型设计的快速认证方法。给定一组基础"训练"提示及其对应的已认证CLIP分类器，OVC基于如下观察：带有新颖提示的分类器可被视为基础训练集中邻近分类器的扰动版本。因此，OVC能够利用增量随机平滑的变体快速认证新颖分类器。通过缓存技巧，我们实现了对新颖提示认证过程约两个数量级的加速。为获得进一步（启发式）加速，OVC使用多元正态分布近似给定输入处的嵌入空间，从而无需通过视觉主干的前向传播进行采样。我们在CIFAR-10和ImageNet测试数据集上使用多种视觉语言主干进行实验评估，验证了OVC的有效性。