Automatically determining whether a text and a corresponding image are semantically aligned is a significant challenge for vision-language models, with applications in generative text-to-image and image-to-text tasks. In this work, we study methods for automatic text-image alignment evaluation. We first introduce SeeTRUE: a comprehensive evaluation set, spanning multiple datasets from both text-to-image and image-to-text generation tasks, with human judgements for whether a given text-image pair is semantically aligned. We then describe two automatic methods to determine alignment: the first involving a pipeline based on question generation and visual question answering models, and the second employing an end-to-end classification approach by finetuning multimodal pretrained models. Both methods surpass prior approaches in various text-image alignment tasks, with significant improvements in challenging cases that involve complex composition or unnatural images. Finally, we demonstrate how our approaches can localize specific misalignments between an image and a given text, and how they can be used to automatically re-rank candidates in text-to-image generation.

翻译：自动判断文本与对应图像是否语义对齐，是视觉-语言模型面临的重要挑战，其应用涵盖生成式文本到图像及图像到文本任务。本研究针对文本-图像自动对齐评估方法展开探索。首先，我们提出了SeeTRUE：一个综合评估数据集，整合了来自文本到图像与图像到文本生成任务的多个数据集，并包含人类对给定文本-图像对语义对齐程度的判断标注。随后，我们描述两种自动对齐判定方法：第一种基于问题生成与视觉问答模型构建流水线，第二种通过微调多模态预训练模型实现端到端分类。两种方法在多种文本-图像对齐任务中均超越现有方案，尤其在涉及复杂构图或非自然图像的挑战性案例中取得显著改进。最后，我们展示了所提方法如何定位图像与给定文本间的具体失配问题，以及如何自动重排序文本到图像生成中的候选结果。