In this work, we investigate the problem of Model-Agnostic Zero-Shot Classification (MA-ZSC), which refers to training non-specific classification architectures (downstream models) to classify real images without using any real images during training. Recent research has demonstrated that generating synthetic training images using diffusion models provides a potential solution to address MA-ZSC. However, the performance of this approach currently falls short of that achieved by large-scale vision-language models. One possible explanation is a potential significant domain gap between synthetic and real images. Our work offers a fresh perspective on the problem by providing initial insights that MA-ZSC performance can be improved by improving the diversity of images in the generated dataset. We propose a set of modifications to the text-to-image generation process using a pre-trained diffusion model to enhance diversity, which we refer to as our $\textbf{bag of tricks}$. Our approach shows notable improvements in various classification architectures, with results comparable to state-of-the-art models such as CLIP. To validate our approach, we conduct experiments on CIFAR10, CIFAR100, and EuroSAT, which is particularly difficult for zero-shot classification due to its satellite image domain. We evaluate our approach with five classification architectures, including ResNet and ViT. Our findings provide initial insights into the problem of MA-ZSC using diffusion models. All code will be available on GitHub.

翻译：本研究探讨了模型无关零样本分类（MA-ZSC）问题，即在训练过程中不使用任何真实图像，训练非特定分类架构（下游模型）对真实图像进行分类。近年研究表明，利用扩散模型生成合成训练图像为解决MA-ZSC提供了潜在方案。然而，该方法当前性能仍不及大规模视觉语言模型。一个可能的解释是合成图像与真实图像之间存在显著领域差距。本研究通过初步论证发现：提升生成数据集中图像的多样性即可改善MA-ZSC性能，为该问题提供了全新视角。我们提出了一套基于预训练扩散模型的文本到图像生成过程改进策略（称为$\textbf{技巧集}$）以增强多样性。该方案在多种分类架构上取得显著提升，性能可媲美CLIP等先进模型。为验证方法有效性，我们在CIFAR10、CIFAR100及卫星图像领域极具挑战性的EuroSAT数据集上开展实验，并采用包含ResNet、ViT在内的五种分类架构进行评估。本研究为基于扩散模型的MA-ZSC问题提供了开创性启示。全部代码将开源至GitHub。