The recent wave of large-scale text-to-image diffusion models has dramatically increased our text-based image generation abilities. These models can generate realistic images for a staggering variety of prompts and exhibit impressive compositional generalization abilities. Almost all use cases thus far have solely focused on sampling; however, diffusion models can also provide conditional density estimates, which are useful for tasks beyond image generation. In this paper, we show that the density estimates from large-scale text-to-image diffusion models like Stable Diffusion can be leveraged to perform zero-shot classification without any additional training. Our generative approach to classification, which we call Diffusion Classifier, attains strong results on a variety of benchmarks and outperforms alternative methods of extracting knowledge from diffusion models. Although a gap remains between generative and discriminative approaches on zero-shot recognition tasks, we find that our diffusion-based approach has stronger multimodal relational reasoning abilities than competing discriminative approaches. Finally, we use Diffusion Classifier to extract standard classifiers from class-conditional diffusion models trained on ImageNet. Even though these models are trained with weak augmentations and no regularization, they approach the performance of SOTA discriminative classifiers. Overall, our results are a step toward using generative over discriminative models for downstream tasks. Results and visualizations at https://diffusion-classifier.github.io/

翻译：近期大规模文本到图像扩散模型浪潮显著提升了我们基于文本生成图像的能力。这些模型能为海量提示生成逼真图像，并展现出惊人的组合泛化能力。迄今为止，几乎所有应用都聚焦于采样环节；然而，扩散模型同样能提供条件密度估计，这对图像生成之外的任务同样具有价值。本文证明，像Stable Diffusion这类大规模文本到图像扩散模型的密度估计可被用于零样本分类，而无需额外训练。我们将这种生成式分类方法命名为扩散分类器，它在多项基准测试中取得了优异结果，且优于从扩散模型中提取知识的替代方案。尽管在零样本识别任务上生成式与判别式方法之间仍存在差距，但我们发现基于扩散的方法具有比竞争性判别式方法更强的多模态关系推理能力。最终，我们运用扩散分类器从基于ImageNet训练的条件扩散模型中提取标准分类器。即使这些模型采用弱增强且未加正则化训练，其性能仍接近最先进的判别式分类器。总体而言，我们的成果标志着在将生成式模型替代判别式模型用于下游任务方面迈出了重要一步。相关结果与可视化内容详见https://diffusion-classifier.github.io/