The excellent generative capabilities of text-to-image diffusion models suggest they learn informative representations of image-text data. However, what knowledge their representations capture is not fully understood, and they have not been thoroughly explored on downstream tasks. We investigate diffusion models by proposing a method for evaluating them as zero-shot classifiers. The key idea is using a diffusion model's ability to denoise a noised image given a text description of a label as a proxy for that label's likelihood. We apply our method to Stable Diffusion and Imagen, using it to probe fine-grained aspects of the models' knowledge and comparing them with CLIP's zero-shot abilities. They perform competitively with CLIP on a wide range of zero-shot image classification datasets. Additionally, they achieve state-of-the-art results on shape/texture bias tests and can successfully perform attribute binding while CLIP cannot. Although generative pre-training is prevalent in NLP, visual foundation models often use other methods such as contrastive learning. Based on our findings, we argue that generative pre-training should be explored as a compelling alternative for vision-language tasks.

翻译：文本到图像扩散模型优越的生成能力表明它们学习到了图像-文本数据中丰富的表示。然而，其表示所捕获的知识尚未被完全理解，并且在下游任务上也未得到深入探索。我们通过提出一种将扩散模型评估为零样本分类器的方法来研究它们。核心思想是利用扩散模型根据标签文本描述对噪声图像进行去噪的能力，作为该标签似然度的代理指标。我们将该方法应用于Stable Diffusion和Imagen，用以探究模型知识的细粒度方面，并将其与CLIP的零样本能力进行比较。在广泛的零样本图像分类数据集上，它们的表现与CLIP相当。此外，它们在形状/纹理偏置测试中取得了最先进的结果，并且能够成功执行属性绑定，而CLIP无法做到。尽管生成式预训练在自然语言处理中普遍存在，但视觉基础模型通常采用对比学习等其他方法。基于我们的发现，我们认为生成式预训练应被探索为视觉-语言任务中一种极具竞争力的替代方案。