Recent advancements in text-to-image (T2I) generative models have shown remarkable capabilities in producing diverse and imaginative visuals based on text prompts. Despite the advancement, these diffusion models sometimes struggle to translate the semantic content from the text into images entirely. While conditioning on the layout has shown to be effective in improving the compositional ability of T2I diffusion models, they typically require manual layout input. In this work, we introduce a novel approach to improving T2I diffusion models using Large Language Models (LLMs) as layout generators. Our method leverages the Chain-of-Thought prompting of LLMs to interpret text and generate spatially reasonable object layouts. The generated layout is then used to enhance the generated images' composition and spatial accuracy. Moreover, we propose an efficient adapter based on a cross-attention mechanism, which explicitly integrates the layout information into the stable diffusion models. Our experiments demonstrate significant improvements in image quality and layout accuracy, showcasing the potential of LLMs in augmenting generative image models.

翻译：文本到图像生成模型的最新进展在基于文本提示生成多样化和富有想象力的视觉内容方面展现了卓越能力。尽管取得了进步，这些扩散模型有时难以将文本中的语义内容完全转化为图像。虽然基于布局的条件控制在提升文本到图像扩散模型的组合能力方面效果显著，但这类方法通常需要手动输入布局。本文提出了一种新颖方法，利用大语言模型作为布局生成器来改进文本到图像扩散模型。我们的方法采用大语言模型的思维链提示机制来解析文本并生成空间合理的物体布局，随后使用生成的布局来提升生成图像的构图与空间准确性。此外，我们基于交叉注意力机制提出了一种高效适配器，将布局信息显式整合到稳定扩散模型中。实验结果表明，该方法在图像质量和布局准确性方面均有显著提升，充分展示了大语言模型增强生成式图像模型的潜力。