Text-conditioned image generation has made significant progress in recent years with generative adversarial networks and more recently, diffusion models. While diffusion models conditioned on text prompts have produced impressive and high-quality images, accurately representing complex text prompts such as the number of instances of a specific object remains challenging. To address this limitation, we propose a novel guidance approach for the sampling process in the diffusion model that leverages bounding box and segmentation map information at inference time without additional training data. Through a novel loss in the sampling process, our approach guides the model with semantic features from CLIP embeddings and enforces geometric constraints, leading to high-resolution images that accurately represent the scene. To obtain bounding box and segmentation map information, we structure the text prompt as a scene graph and enrich the nodes with CLIP embeddings. Our proposed model achieves state-of-the-art performance on two public benchmarks for image generation from scene graphs, surpassing both scene graph to image and text-based diffusion models in various metrics. Our results demonstrate the effectiveness of incorporating bounding box and segmentation map guidance in the diffusion model sampling process for more accurate text-to-image generation.

翻译：文本条件图像生成在近年来取得了显著进展，这得益于生成对抗网络以及更近期的扩散模型。尽管基于文本提示的扩散模型生成了令人印象深刻的高质量图像，但在准确表示复杂文本提示（如特定对象的实例数量）方面仍存在挑战。为解决此限制，我们提出了一种适用于扩散模型采样过程的新型引导方法，该方法在推理时利用边界框和分割图信息，无需额外训练数据。通过在采样过程中引入新颖的损失函数，我们的方法利用来自CLIP嵌入的语义特征引导模型，并施加几何约束，从而生成能够准确表示场景的高分辨率图像。为获取边界框和分割图信息，我们将文本提示构建为场景图，并用CLIP嵌入丰富节点信息。所提出的模型在两个用于场景图生成图像的公开基准数据集上达到了最先进的性能，在多种指标上超越了场景图到图像以及基于文本的扩散模型。我们的结果证明了在扩散模型采样过程中融入边界框和分割图引导能够有效提升文本到图像生成的准确性。