Recent advances in text-to-image diffusion models have enabled the generation of diverse and high-quality images. However, generated images often fall short of depicting subtle details and are susceptible to errors due to ambiguity in the input text. One way of alleviating these issues is to train diffusion models on class-labeled datasets. This comes with a downside, doing so limits their expressive power: (i) supervised datasets are generally small compared to large-scale scraped text-image datasets on which text-to-image models are trained, and so the quality and diversity of generated images are severely affected, or (ii) the input is a hard-coded label, as opposed to free-form text, which limits the control over the generated images. In this work, we propose a non-invasive fine-tuning technique that capitalizes on the expressive potential of free-form text while achieving high accuracy through discriminative signals from a pretrained classifier, which guides the generation. This is done by iteratively modifying the embedding of a single input token of a text-to-image diffusion model, using the classifier, by steering generated images toward a given target class. Our method is fast compared to prior fine-tuning methods and does not require a collection of in-class images or retraining of a noise-tolerant classifier. We evaluate our method extensively, showing that the generated images are: (i) more accurate and of higher quality than standard diffusion models, (ii) can be used to augment training data in a low-resource setting, and (iii) reveal information about the data used to train the guiding classifier. The code is available at \url{https://github.com/idansc/discriminative_class_tokens}

翻译：近年来，文本到图像扩散模型的进展使得生成多样且高质量的图像成为可能。然而，生成的图像往往在描绘细微细节方面有所欠缺，且容易因输入文本的歧义而产生错误。缓解这些问题的一种方法是在类别标记数据集上训练扩散模型。但这会带来一个弊端：这种做法限制了模型的表达能力：（i）与训练文本到图像模型所依赖的大规模爬取文本-图像数据集相比，监督数据集通常较小，因此生成图像的质量和多样性会受到严重影响；（ii）输入是硬编码的标签，而非自由形式的文本，这限制了对生成图像的控制。在本工作中，我们提出了一种非侵入式微调技术，该技术利用自由形式文本的表达潜力，同时通过预训练分类器的判别性信号实现高精度，从而引导生成过程。具体而言，我们通过分类器迭代修改文本到图像扩散模型中单个输入标记的嵌入向量，引导生成图像朝向给定的目标类别。与先前的微调方法相比，我们的方法运行速度快，且无需收集同类别图像集或重新训练噪声容忍分类器。我们对该方法进行了全面评估，结果表明生成的图像：（i）比标准扩散模型更准确且质量更高；（ii）可用于在低资源场景下扩充训练数据；（iii）能揭示用于训练引导分类器的数据信息。代码发布于\url{https://github.com/idansc/discriminative_class_tokens}。