This paper presents novel benchmarks for evaluating vision-language models (VLMs) in zero-shot recognition, focusing on granularity and specificity. Although VLMs excel in tasks like image captioning, they face challenges in open-world settings. Our benchmarks test VLMs' consistency in understanding concepts across semantic granularity levels and their response to varying text specificity. Findings show that VLMs favor moderately fine-grained concepts and struggle with specificity, often misjudging texts that differ from their training data. Extensive evaluations reveal limitations in current VLMs, particularly in distinguishing between correct and subtly incorrect descriptions. While fine-tuning offers some improvements, it doesn't fully address these issues, highlighting the need for VLMs with enhanced generalization capabilities for real-world applications. This study provides insights into VLM limitations and suggests directions for developing more robust models.

翻译：本文提出了评估视觉语言模型在零样本识别任务中表现的新基准，重点关注粒度与特异性两个维度。尽管视觉语言模型在图像描述等任务中表现出色，但在开放世界场景中仍面临挑战。我们的基准测试检验了视觉语言模型在理解不同语义粒度层级概念时的一致性，以及其对文本特异性变化的响应能力。研究发现，视觉语言模型倾向于中等细粒度的概念，且在特异性处理上存在困难，经常对与训练数据存在差异的文本产生误判。大量评估揭示了当前视觉语言模型的局限性，特别是在区分正确描述与细微错误描述方面。虽然微调能带来一定改进，但未能完全解决这些问题，凸显了开发具备更强泛化能力的视觉语言模型对于实际应用的必要性。本研究深入剖析了视觉语言模型的局限，并为开发更鲁棒的模型指明了方向。