Zero-shot learning in prompted vision-language models, the practice of crafting prompts to build classifiers without an explicit training process, has achieved impressive performance in many settings. This success presents a seemingly surprising observation: these methods suffer relatively little from overfitting, i.e., when a prompt is manually engineered to achieve low error on a given training set (thus rendering the method no longer actually zero-shot), the approach still performs well on held-out test data. In this paper, we show that we can explain such performance well via recourse to classical PAC-Bayes bounds. Specifically, we show that the discrete nature of prompts, combined with a PAC-Bayes prior given by a language model, results in generalization bounds that are remarkably tight by the standards of the literature: for instance, the generalization bound of an ImageNet classifier is often within a few percentage points of the true test error. We demonstrate empirically that this holds for existing handcrafted prompts and prompts generated through simple greedy search. Furthermore, the resulting bound is well-suited for model selection: the models with the best bound typically also have the best test performance. This work thus provides a possible justification for the widespread practice of prompt engineering, even if it seems that such methods could potentially overfit the training data.

翻译：零样本学习在提示式视觉-语言模型中，即通过设计提示来构建分类器而无需显式训练过程，已在多种场景中取得了令人瞩目的表现。这一成功呈现出一个看似令人惊讶的观察：这些方法相对较少受到过拟合的影响，即当通过手动设计提示在给定训练集上实现低误差（从而使该方法不再真正为零样本）时，该方式在保留测试数据上仍表现良好。本文表明，我们可以通过借助经典的PAC-Bayes界限很好地解释这种性能。具体而言，我们证明，提示的离散性质与语言模型给出的PAC-Bayes先验相结合，得到的泛化界限在文献标准中极为紧凑：例如，ImageNet分类器的泛化界限通常与真实测试误差相差仅几个百分点。我们通过实验证明，这一结论对现有手工设计的提示以及通过简单贪心搜索生成的提示均成立。此外，由此得出的界限非常适合模型选择：具有最佳界限的模型通常也拥有最佳测试性能。因此，这项工作为提示工程的广泛实践提供了可能的合理性解释，即便此类方法似乎可能对训练数据产生过拟合。