Text-to-image diffusion models can synthesize a large variety of concepts in new compositions and scenarios. However, they still struggle with generating uncommon concepts, rare unusual combinations, or structured concepts like hand palms. Their limitation is partly due to the long-tail nature of their training data: web-crawled data sets are strongly unbalanced, causing models to under-represent concepts from the tail of the distribution. Here we characterize the effect of unbalanced training data on text-to-image models and offer a remedy. We show that rare concepts can be correctly generated by carefully selecting suitable generation seeds in the noise space, a technique that we call SeedSelect. SeedSelect is efficient and does not require retraining the diffusion model. We evaluate the benefit of SeedSelect on a series of problems. First, in few-shot semantic data augmentation, where we generate semantically correct images for few-shot and long-tail benchmarks. We show classification improvement on all classes, both from the head and tail of the training data of diffusion models. We further evaluate SeedSelect on correcting images of hands, a well-known pitfall of current diffusion models, and show that it improves hand generation substantially.

翻译：文本到图像扩散模型能够在新组合和场景中合成大量概念。然而，它们在生成罕见概念、罕见异常组合或结构化概念（如手掌）时仍面临挑战。这一局限性部分源于其训练数据的长尾特性：网络爬取的数据集严重不平衡，导致模型对分布尾部概念的呈现不足。本文刻画了不平衡训练数据对文本到图像模型的影响，并提出一种补救方法。我们证明，通过在噪声空间中精心选择生成种子，可以正确生成罕见概念，我们将该技术称为SeedSelect。SeedSelect高效且无需重新训练扩散模型。我们在一系列问题上评估了SeedSelect的优势。首先，在少样本语义数据增强任务中，我们为少样本和长尾基准生成了语义正确的图像。结果显示，所有类别（包括扩散模型训练数据的头部和尾部类别）的分类性能均得到提升。我们还针对当前扩散模型的一个已知缺陷——手部图像修正——评估了SeedSelect，并表明它能显著改善手部生成效果。