The recent wave of large-scale text-to-image diffusion models has dramatically increased our text-based image generation abilities. These models can generate realistic images for a staggering variety of prompts and exhibit impressive compositional generalization abilities. Almost all use cases thus far have solely focused on sampling; however, diffusion models can also provide conditional density estimates, which are useful for tasks beyond image generation. In this paper, we show that the density estimates from large-scale text-to-image diffusion models like Stable Diffusion can be leveraged to perform zero-shot classification without any additional training. Our generative approach to classification, which we call Diffusion Classifier, attains strong results on a variety of benchmarks and outperforms alternative methods of extracting knowledge from diffusion models. Although a gap remains between generative and discriminative approaches on zero-shot recognition tasks, our diffusion-based approach has significantly stronger multimodal compositional reasoning ability than competing discriminative approaches. Finally, we use Diffusion Classifier to extract standard classifiers from class-conditional diffusion models trained on ImageNet. Our models achieve strong classification performance using only weak augmentations and exhibit qualitatively better "effective robustness" to distribution shift. Overall, our results are a step toward using generative over discriminative models for downstream tasks. Results and visualizations at https://diffusion-classifier.github.io/

翻译：近年来，大规模文本到图像扩散模型的浪潮显著提升了我们基于文本生成图像的能力。这些模型能够针对海量多样化提示生成逼真图像，并展现出惊人的组合泛化能力。迄今为止，几乎所有应用案例都聚焦于采样环节；然而，扩散模型还能提供条件密度估计，这对于图像生成之外的任务同样有用。在本文中，我们展示了像Stable Diffusion这样的大规模文本到图像扩散模型产生的密度估计，可以在无需额外训练的情况下用于零样本分类。我们将这种生成式分类方法称为扩散分类器（Diffusion Classifier），它在多种基准测试上取得了强劲结果，并优于从扩散模型中提取知识的其他替代方法。尽管在零样本识别任务上，生成式方法与判别式方法之间仍存在差距，但我们的基于扩散的方法在多模态组合推理能力上显著强于竞争的判别式方法。最后，我们利用扩散分类器从基于ImageNet训练的条件扩散模型中提取标准分类器。我们的模型仅使用弱增强就实现了强大的分类性能，并在分布偏移下展现出定性上更好的“有效鲁棒性”。总体而言，我们的结果向使用生成式模型替代判别式模型用于下游任务迈出了一步。结果和可视化内容见 https://diffusion-classifier.github.io/