Large vision-language models (VLMs), such as CLIP, learn rich joint image-text representations, facilitating advances in numerous downstream tasks, including zero-shot classification and text-to-image generation. Nevertheless, existing VLMs exhibit a prominent well-documented limitation - they fail to encapsulate compositional concepts such as counting. We introduce a simple yet effective method to improve the quantitative understanding of VLMs, while maintaining their overall performance on common benchmarks. Specifically, we propose a new counting-contrastive loss used to finetune a pre-trained VLM in tandem with its original objective. Our counting loss is deployed over automatically-created counterfactual examples, each consisting of an image and a caption containing an incorrect object count. For example, an image depicting three dogs is paired with the caption "Six dogs playing in the yard". Our loss encourages discrimination between the correct caption and its counterfactual variant which serves as a hard negative example. To the best of our knowledge, this work is the first to extend CLIP's capabilities to object counting. Furthermore, we introduce "CountBench" - a new image-text counting benchmark for evaluating a model's understanding of object counting. We demonstrate a significant improvement over state-of-the-art baseline models on this task. Finally, we leverage our count-aware CLIP model for image retrieval and text-conditioned image generation, demonstrating that our model can produce specific counts of objects more reliably than existing ones.

翻译：大型视觉语言模型（VLMs），如CLIP，通过学习丰富的联合图像-文本表示，推动了零样本分类和文本到图像生成等多个下游任务的进步。然而，现有VLM存在一个显著的、被广泛记录的限制——它们难以处理诸如计数之类的组合概念。我们提出了一种简单而有效的方法，旨在提升VLM的定量理解能力，同时保持其在常见基准测试上的整体性能。具体而言，我们提出了一种新的计数对比损失函数，用于与原始目标函数协同微调预训练的VLM。该计数损失基于自动生成的假设反事实示例进行部署，每个示例包含一张图像和一个描述物体数量错误的标题。例如，一张显示三只狗的图像被配以标题“六只狗在院子里玩耍”。该损失函数鼓励模型区分正确标题与其作为困难负例的反事实变体。据我们所知，这项工作首次将CLIP的能力扩展至物体计数。此外，我们引入了“CountBench”——一个新的用于评估模型物体计数理解的图像-文本计数基准。我们在此任务上展示了相较于最先进基线模型的显著改进。最后，我们利用具备计数感知能力的CLIP模型进行图像检索和文本条件图像生成，证明我们的模型能够比现有模型更可靠地生成特定数量的物体。