In this work, we investigate the problem of Model-Agnostic Zero-Shot Classification (MA-ZSC), which refers to training non-specific classification architectures (downstream models) to classify real images without using any real images during training. Recent research has demonstrated that generating synthetic training images using diffusion models provides a potential solution to address MA-ZSC. However, the performance of this approach currently falls short of that achieved by large-scale vision-language models. One possible explanation is a potential significant domain gap between synthetic and real images. Our work offers a fresh perspective on the problem by providing initial insights that MA-ZSC performance can be improved by improving the diversity of images in the generated dataset. We propose a set of modifications to the text-to-image generation process using a pre-trained diffusion model to enhance diversity, which we refer to as our $\textbf{bag of tricks}$. Our approach shows notable improvements in various classification architectures, with results comparable to state-of-the-art models such as CLIP. To validate our approach, we conduct experiments on CIFAR10, CIFAR100, and EuroSAT, which is particularly difficult for zero-shot classification due to its satellite image domain. We evaluate our approach with five classification architectures, including ResNet and ViT. Our findings provide initial insights into the problem of MA-ZSC using diffusion models. All code will be available on GitHub.

翻译：本研究探讨了模型无关零样本分类（MA-ZSC）问题，即在训练过程中无需使用任何真实图像，训练非特定分类架构（下游模型）对真实图像进行分类。近年研究表明，利用扩散模型生成合成训练图像为解决MA-ZSC提供了潜在方案。然而，该方法的性能目前仍不及大规模视觉语言模型。一个可能的原因是合成图像与真实图像之间存在显著域差距。本工作通过初步洞察指出，通过提升生成数据集中图像的多样性可改善MA-ZSC性能，从而为该问题提供了全新视角。我们提出了一系列针对预训练扩散模型文本到图像生成过程的改进方法（命名为$\textbf{技巧包}$）以增强多样性。我们的方法在多种分类架构上取得了显著提升，其性能与CLIP等最先进模型相当。为验证方法有效性，我们在CIFAR10、CIFAR100以及因卫星图像域而极具零样本分类难度的EuroSAT数据集上开展实验。我们采用包括ResNet和ViT在内的五种分类架构进行评估。研究结果为基于扩散模型的MA-ZSC问题提供了初步洞见。所有代码将在GitHub上开源。