Text-conditioned image generation models have recently shown immense qualitative success using denoising diffusion processes. However, unlike discriminative vision-and-language models, it is a non-trivial task to subject these diffusion-based generative models to automatic fine-grained quantitative evaluation of high-level phenomena such as compositionality. Towards this goal, we perform two innovations. First, we transform diffusion-based models (in our case, Stable Diffusion) for any image-text matching (ITM) task using a novel method called DiffusionITM. Second, we introduce the Generative-Discriminative Evaluation Benchmark (GDBench) benchmark with 7 complex vision-and-language tasks, bias evaluation and detailed analysis. We find that Stable Diffusion + DiffusionITM is competitive on many tasks and outperforms CLIP on compositional tasks like like CLEVR and Winoground. We further boost its compositional performance with a transfer setup by fine-tuning on MS-COCO while retaining generative capabilities. We also measure the stereotypical bias in diffusion models, and find that Stable Diffusion 2.1 is, for the most part, less biased than Stable Diffusion 1.5. Overall, our results point in an exciting direction bringing discriminative and generative model evaluation closer. We will release code and benchmark setup soon.

翻译：文本条件图像生成模型近期通过去噪扩散过程展现了惊人的定性生成质量。然而与判别式视觉-语言模型不同，对这类基于扩散的生成模型进行细粒度自动定量评估（如组合性等高阶现象）并非易事。为实现这一目标，我们提出两项创新：首先，采用名为DiffusionITM的新方法将扩散模型（本文使用Stable Diffusion）转化为适用于任意图像-文本匹配任务的模型；其次，构建包含7个复杂视觉-语言任务、偏见评估及详细分析的生成-判别评估基准（GDBench）。实验表明，Stable Diffusion结合DiffusionITM在多数任务中表现优异，且在CLEVR和Winoground等组合性任务上超越CLIP。通过MS-COCO微调的迁移学习方案，我们在保持生成能力的同时进一步提升了组合性表现。我们对扩散模型中的刻板偏见进行量化分析，发现Stable Diffusion 2.1整体上较1.5版本具有更低偏见。总体而言，本研究为拉近判别式与生成式模型评估距离提供了令人振奋的研究方向。相关代码与基准配置即将开源。