A key benefit of deep vision-language models such as CLIP is that they enable zero-shot open vocabulary classification; the user has the ability to define novel class labels via natural language prompts at inference time. However, while CLIP-based zero-shot classifiers have demonstrated competitive performance across a range of domain shifts, they remain highly vulnerable to adversarial attacks. Therefore, ensuring the robustness of such models is crucial for their reliable deployment in the wild. In this work, we introduce Open Vocabulary Certification (OVC), a fast certification method designed for open-vocabulary models like CLIP via randomized smoothing techniques. Given a base "training" set of prompts and their corresponding certified CLIP classifiers, OVC relies on the observation that a classifier with a novel prompt can be viewed as a perturbed version of nearby classifiers in the base training set. Therefore, OVC can rapidly certify the novel classifier using a variation of incremental randomized smoothing. By using a caching trick, we achieve approximately two orders of magnitude acceleration in the certification process for novel prompts. To achieve further (heuristic) speedups, OVC approximates the embedding space at a given input using a multivariate normal distribution bypassing the need for sampling via forward passes through the vision backbone. We demonstrate the effectiveness of OVC on through experimental evaluation using multiple vision-language backbones on the CIFAR-10 and ImageNet test datasets.

翻译：深度视觉语言模型（如CLIP）的核心优势在于支持零样本开放词汇分类——用户可在推理时通过自然语言提示自主定义新类别标签。然而，尽管基于CLIP的零样本分类器在各类域迁移场景中展现出竞争性表现，其仍极易遭受对抗攻击。因此，确保此类模型的鲁棒性对野外部署的可靠性至关重要。本研究提出开放词汇认证（OVC），一种专为CLIP等开放词汇模型设计的快速认证方法。通过随机平滑技术，OVC基于基础"训练"提示集及其对应的已认证CLIP分类器，观察到含新提示的分类器可视为基础训练集中邻近分类器的扰动版本。由此，OVC可采用增量随机平滑变体快速认证新分类器。通过缓存技巧，我们实现了对新提示认证过程约两个数量级的加速。为获得启发式加速，OVC利用多元正态分布近似给定输入下的嵌入空间，从而避免通过视觉骨干网络进行前向传播采样。通过在CIFAR-10和ImageNet测试数据集上使用多种视觉语言骨干网络的实验评估，我们验证了OVC的有效性。