Recent large vision-language models such as CLIP have shown remarkable out-of-distribution (OOD) detection and generalization performance. However, their zero-shot in-distribution (ID) accuracy is often limited for downstream datasets. Recent CLIP-based fine-tuning methods such as prompt learning have demonstrated significant improvements in ID classification and OOD generalization where OOD labels are available. Nonetheless, it remains unclear whether the model is reliable to semantic shifts without OOD labels. In this paper, we aim to bridge the gap and present a comprehensive study to understand how fine-tuning impact OOD detection for few-shot downstream tasks. By framing OOD detection as multi-modal concept matching, we establish a connection between fine-tuning methods and various OOD scores. Our results suggest that a proper choice of OOD scores is essential for CLIP-based fine-tuning. In particular, the maximum concept matching (MCM) score provides a promising solution consistently. We also show that prompt learning demonstrates the state-of-the-art OOD detection performance over the zero-shot counterpart.

翻译：近期如CLIP等大规模视觉语言模型在分布外（OOD）检测和泛化性能上表现卓越。然而，其零样本分布内（ID）准确率在面向下游数据集时往往受限。基于CLIP的近期微调方法（如提示学习）在利用已知OOD标签进行ID分类和OOD泛化方面展现了显著改进。尽管如此，当缺乏OOD标签时，模型能否可靠应对语义偏移仍不明确。本文旨在填补这一空白，通过系统性研究阐明微调如何影响小样本下游任务的OOD检测性能。通过将OOD检测建模为多模态概念匹配，我们建立了微调方法与各类OOD评分之间的关联。研究结果表明，合理选择OOD评分对基于CLIP的微调至关重要，其中最大概念匹配（MCM）评分始终提供有前景的解决方案。我们还证明，提示学习在OOD检测性能上相较零样本基线展现出最先进水平。