We introduce SPARse Fine-grained Contrastive Alignment (SPARC), a simple method for pretraining more fine-grained multimodal representations from image-text pairs. Given that multiple image patches often correspond to single words, we propose to learn a grouping of image patches for every token in the caption. To achieve this, we use a sparse similarity metric between image patches and language tokens and compute for each token a language-grouped vision embedding as the weighted average of patches. The token and language-grouped vision embeddings are then contrasted through a fine-grained sequence-wise loss that only depends on individual samples and does not require other batch samples as negatives. This enables more detailed information to be learned in a computationally inexpensive manner. SPARC combines this fine-grained loss with a contrastive loss between global image and text embeddings to learn representations that simultaneously encode global and local information. We thoroughly evaluate our proposed method and show improved performance over competing approaches both on image-level tasks relying on coarse-grained information, e.g. classification, as well as region-level tasks relying on fine-grained information, e.g. retrieval, object detection, and segmentation. Moreover, SPARC improves model faithfulness and captioning in foundational vision-language models.

翻译：我们提出SPARse细粒度对比对齐（SPARC），一种从图像-文本对中预训练更细粒度多模态表示的简单方法。鉴于多个图像块通常对应单个单词，我们提出为描述中的每个词元学习一组图像块分组。为此，我们利用图像块与语言词元之间的稀疏相似性度量，并为每个词元计算语言分组的视觉嵌入，即图像块的加权平均。随后，通过仅依赖于单个样本且无需其他批样本作为负样本的细粒度序列级损失，对词元与语言分组视觉嵌入进行对比。这使得能够以计算代价较低的方式学习更详细的信息。SPARC将该细粒度损失与全局图像和文本嵌入之间的对比损失相结合，以学习同时编码全局和局部信息的表示。我们全面评估了所提方法，并在依赖粗粒度信息的图像级任务（如分类）以及依赖细粒度信息的区域级任务（如检索、目标检测和分割）上，展示了相较于竞争方法的性能提升。此外，SPARC提升了基础视觉-语言模型的模型忠实度和描述生成能力。