Recent advances in text-to-image diffusion models have enabled the generation of diverse and high-quality images. While impressive, the images often fall short of depicting subtle details and are susceptible to errors due to ambiguity in the input text. One way of alleviating these issues is to train diffusion models on class-labeled datasets. This approach has two disadvantages: (i) supervised datasets are generally small compared to large-scale scraped text-image datasets on which text-to-image models are trained, affecting the quality and diversity of the generated images, or (ii) the input is a hard-coded label, as opposed to free-form text, limiting the control over the generated images. In this work, we propose a non-invasive fine-tuning technique that capitalizes on the expressive potential of free-form text while achieving high accuracy through discriminative signals from a pretrained classifier. This is done by iteratively modifying the embedding of an added input token of a text-to-image diffusion model, by steering generated images toward a given target class according to a classifier. Our method is fast compared to prior fine-tuning methods and does not require a collection of in-class images or retraining of a noise-tolerant classifier. We evaluate our method extensively, showing that the generated images are: (i) more accurate and of higher quality than standard diffusion models, (ii) can be used to augment training data in a low-resource setting, and (iii) reveal information about the data used to train the guiding classifier. The code is available at \url{https://github.com/idansc/discriminative_class_tokens}.

翻译：近期文本到图像扩散模型的进展使得生成多样化且高质量的图像成为可能。尽管效果显著，但这些图像往往在刻画细微细节方面有所欠缺，且容易因输入文本的歧义性而产生错误。缓解这些问题的一种方法是基于类别标记的数据集训练扩散模型。然而，该方法存在两个缺陷：（i）与文本到图像模型所依赖的大规模网络抓取文本-图像数据集相比，监督数据集通常规模较小，从而影响生成图像的质量与多样性；（ii）输入为硬编码标签而非自由格式文本，限制了用户对生成图像的控制能力。在本工作中，我们提出一种非侵入式微调技术，该技术既充分利用自由格式文本的表达潜力，又能通过预训练分类器的判别性信号实现高准确度。我们通过迭代修改文本到图像扩散模型中新增输入标记的嵌入向量，使生成图像依据分类器向目标类别偏移。与现有微调方法相比，我们的方法速度更快，且无需收集同类图像或重新训练抗噪声分类器。大量实验表明，该方法生成的图像具有以下特性：（i）比标准扩散模型更准确、质量更高；（ii）可用于低资源场景下的训练数据增强；（iii）能够揭示引导分类器训练数据的相关信息。代码已开源至\url{https://github.com/idansc/discriminative_class_tokens}。