Text-to-image synthesis has recently attracted widespread attention due to rapidly improving quality and numerous practical applications. However, the language understanding capabilities of text-to-image models are still poorly understood, which makes it difficult to reason about prompt formulations that a given model would understand well. In this work, we measure the capability of popular text-to-image models to understand $\textit{hypernymy}$, or the "is-a" relation between words. We design two automatic metrics based on the WordNet semantic hierarchy and existing image classifiers pretrained on ImageNet. These metrics both enable broad quantitative comparison of linguistic capabilities for text-to-image models and offer a way of finding fine-grained qualitative differences, such as words that are unknown to models and thus are difficult for them to draw. We comprehensively evaluate popular text-to-image models, including GLIDE, Latent Diffusion, and Stable Diffusion, showing how our metrics can provide a better understanding of the individual strengths and weaknesses of these models.

翻译：文本到图像合成技术因图像质量的快速提升及众多实际应用而近期受到广泛关注。然而，当前对文本到图像模型的语言理解能力仍知之甚少，这导致难以推断特定模型能良好理解的提示表述。本研究衡量了主流文本到图像模型对$\textit{超义关系}$（即词语间的“是一种”关系）的理解能力。我们基于WordNet语义层级结构及在ImageNet上预训练的现有图像分类器设计了两项自动评估指标。这两项指标既能对文本到图像模型的语言能力进行广泛的定量比较，也能揭示细粒度的定性差异，例如模型无法识别的词语（因此难以将其绘制）。我们全面评估了GLIDE、潜在扩散模型及稳定扩散等主流文本到图像模型，展示了我们的指标如何有助于更深入理解这些模型的各自优势与不足。