The ability to understand visual concepts and replicate and compose these concepts from images is a central goal for computer vision. Recent advances in text-to-image (T2I) models have lead to high definition and realistic image quality generation by learning from large databases of images and their descriptions. However, the evaluation of T2I models has focused on photorealism and limited qualitative measures of visual understanding. To quantify the ability of T2I models in learning and synthesizing novel visual concepts, we introduce ConceptBed, a large-scale dataset that consists of 284 unique visual concepts, 5K unique concept compositions, and 33K composite text prompts. Along with the dataset, we propose an evaluation metric, Concept Confidence Deviation (CCD), that uses the confidence of oracle concept classifiers to measure the alignment between concepts generated by T2I generators and concepts contained in ground truth images. We evaluate visual concepts that are either objects, attributes, or styles, and also evaluate four dimensions of compositionality: counting, attributes, relations, and actions. Our human study shows that CCD is highly correlated with human understanding of concepts. Our results point to a trade-off between learning the concepts and preserving the compositionality which existing approaches struggle to overcome.

翻译：理解视觉概念并从图像中复制与组合这些概念是计算机视觉的核心目标。近期文本到图像（T2I）模型的进展通过从大规模图像及其描述数据库中学习，实现了高分辨率且逼真的图像生成。然而，T2I模型的评估主要集中在逼真度及有限的视觉理解定性度量上。为量化T2I模型学习与合成新颖视觉概念的能力，我们提出ConceptBed——一个大规模数据集，包含284个独特视觉概念、5000个独特概念组合及33,000条复合文本提示。伴随数据集，我们提出评估指标——概念置信偏差（CCD），该指标利用oracle概念分类器的置信度，衡量T2I生成器生成的概念与真实图像中包含概念之间的一致性。我们评估的对象、属性或风格等视觉概念，同时涵盖计数、属性、关系与动作四个组合维度。人类研究表明，CCD与人类对概念的理解高度相关。实验结果揭示了现有方法在概念学习与保持组合性之间难以兼顾的权衡关系。