Evaluating text-to-image models is notoriously difficult. A strong recent approach for assessing text-image faithfulness is based on QG/A (question generation and answering), which uses pre-trained foundational models to automatically generate a set of questions and answers from the prompt, and output images are scored based on whether these answers extracted with a visual question answering model are consistent with the prompt-based answers. This kind of evaluation is naturally dependent on the quality of the underlying QG and QA models. We identify and address several reliability challenges in existing QG/A work: (a) QG questions should respect the prompt (avoiding hallucinations, duplications, and omissions) and (b) VQA answers should be consistent (not asserting that there is no motorcycle in an image while also claiming the motorcycle is blue). We address these issues with Davidsonian Scene Graph (DSG), an empirically grounded evaluation framework inspired by formal semantics, which is adaptable to any QG/A frameworks. DSG produces atomic and unique questions organized in dependency graphs, which (i) ensure appropriate semantic coverage and (ii) sidestep inconsistent answers. With extensive experimentation and human evaluation on a range of model configurations (LLM, VQA, and T2I), we empirically demonstrate that DSG addresses the challenges noted above. Finally, we present DSG-1k, an open-sourced evaluation benchmark that includes 1,060 prompts, covering a wide range of fine-grained semantic categories with a balanced distribution. We release the DSG-1k prompts and the corresponding DSG questions.

翻译：评估文本到图像模型历来极具挑战性。近期一种基于QG/A（问题生成与回答）的可靠方法被用于评估文本-图像一致性，该方法利用预训练基础模型自动从提示中生成一组问题与答案，并通过视觉问答模型提取的答案与基于提示的答案是否一致来对输出图像进行评分。此类评估自然依赖于底层QG与QA模型的质量。我们识别并解决了现有QG/A工作中的若干可靠性挑战：（a）QG生成的问题应严格遵循提示（避免幻觉、重复和遗漏）；（b）VQA生成的答案应保持一致性（不得同时声称图像中不存在摩托车与摩托车是蓝色）。针对这些问题，我们提出戴维森场景图（DSG）——一种受形式语义学启发、基于经验验证的评估框架，可适配任意QG/A框架。DSG生成以依赖图形式组织的原子化唯一性问题，从而（i）确保适当的语义覆盖范围并（ii）规避不一致的答案。通过对多种模型配置（LLM、VQA及T2I）的广泛实验与人工评估，我们实证表明DSG成功解决了上述挑战。最终，我们发布了DSG-1k端开源评估基准，包含1,060个覆盖广泛细粒度语义类别且分布均衡的提示。我们同时公开了DSG-1k提示及其对应的DSG问题。