Recent large vision-language models such as CLIP have shown remarkable out-of-distribution (OOD) detection and generalization performance. However, their zero-shot in-distribution (ID) accuracy is often limited for downstream datasets. Recent CLIP-based fine-tuning methods such as prompt learning have demonstrated significant improvements in ID classification and OOD generalization where OOD labels are available. Nonetheless, it remains unclear whether the model is reliable to semantic shifts without OOD labels. In this paper, we aim to bridge the gap and present a comprehensive study to understand how fine-tuning impact OOD detection for few-shot downstream tasks. By framing OOD detection as multi-modal concept matching, we establish a connection between fine-tuning methods and various OOD scores. Our results suggest that a proper choice of OOD scores is essential for CLIP-based fine-tuning. In particular, the maximum concept matching (MCM) score provides a promising solution consistently. We also show that prompt learning demonstrates the state-of-the-art OOD detection performance over the zero-shot counterpart.

翻译：近期如CLIP等大型视觉语言模型在分布外检测和泛化性能方面表现卓越。然而，对于下游数据集，其零样本分布内准确率通常受限。基于CLIP的微调方法（如提示学习）在分布内分类和可利用分布外标签的分布外泛化方面取得了显著改进。但缺乏分布外标签时，模型对语义偏移的可靠性仍不清楚。本文旨在弥合这一研究空白，通过全面研究揭示微调如何影响少样本下游任务的分布外检测。将分布外检测建模为多模态概念匹配后，我们建立了微调方法与各类分布外评分之间的关联。实验结果表明，选择合适的分布外评分对基于CLIP的微调至关重要，其中最大概念匹配评分持续提供可靠的解决方案。我们还证明，提示学习在分布外检测性能上优于零样本基线方法，达到了当前最优水平。