Over the past year, a large body of multimodal research has emerged around zero-shot evaluation using GPT descriptors. These studies boost the zero-shot accuracy of pretrained VL models with an ensemble of label-specific text generated by GPT. A recent study, WaffleCLIP, demonstrated that similar zero-shot accuracy can be achieved with an ensemble of random descriptors. However, both zero-shot methods are un-trainable and consequently sub-optimal when some few-shot out-of-distribution (OOD) training data is available. Inspired by these prior works, we present two more flexible methods called descriptor and word soups, which do not require an LLM at test time and can leverage training data to increase OOD target accuracy. Descriptor soup greedily selects a small set of textual descriptors using generic few-shot training data, then calculates robust class embeddings using the selected descriptors. Word soup greedily assembles a chain of words in a similar manner. Compared to existing few-shot soft prompt tuning methods, word soup requires fewer parameters by construction and less GPU memory, since it does not require backpropagation. Both soups outperform current published few-shot methods, even when combined with SoTA zero-shot methods, on cross-dataset and domain generalization benchmarks. Compared with SoTA prompt and descriptor ensembling methods, such as ProDA and WaffleCLIP, word soup achieves higher OOD accuracy with fewer ensemble members. Please checkout our code: github.com/Chris210634/word_soups

翻译：过去一年，大量多模态研究围绕使用GPT描述符进行零样本评估展开。这些研究通过整合由GPT生成的标签特定文本集合，提升了预训练视觉语言模型的零样本准确率。近期研究WaffleCLIP表明，使用随机描述符集合也能达到类似的零样本准确率。然而，这两种零样本方法均不可训练，因此在获得少量分布外训练数据时效果欠佳。受这些前期研究启发，我们提出了两种更灵活的方法——描述符汤和词语汤，这两种方法在测试时无需大语言模型，并能利用训练数据提升分布外目标准确率。描述符汤通过通用少样本训练数据贪婪选取少量文本描述符，随后利用选定描述符计算鲁棒的类别嵌入。词语汤则以类似方式贪婪构建词语链。相较于现有少样本软提示微调方法，词语汤因无需反向传播，在结构上所需参数更少且占用GPU内存更低。在跨数据集和域泛化基准测试中，这两种汤方法均优于当前已发表的少样本方法，即便与最先进的零样本方法结合使用时亦是如此。与ProDA和WaffleCLIP等最先进的提示与描述符集成方法相比，词语汤能用更少的集成成员实现更高的分布外准确率。请查阅我们的代码：github.com/Chris210634/word_soups