Diffusion models have opened up new avenues for the field of image generation, resulting in the proliferation of high-quality models shared on open-source platforms. However, a major challenge persists in current text-to-image systems are often unable to handle diverse inputs, or are limited to single model results. Current unified attempts often fall into two orthogonal aspects: i) parse Diverse Prompts in input stage; ii) activate expert model to output. To combine the best of both worlds, we propose DiffusionGPT, which leverages Large Language Models (LLM) to offer a unified generation system capable of seamlessly accommodating various types of prompts and integrating domain-expert models. DiffusionGPT constructs domain-specific Trees for various generative models based on prior knowledge. When provided with an input, the LLM parses the prompt and employs the Trees-of-Thought to guide the selection of an appropriate model, thereby relaxing input constraints and ensuring exceptional performance across diverse domains. Moreover, we introduce Advantage Databases, where the Tree-of-Thought is enriched with human feedback, aligning the model selection process with human preferences. Through extensive experiments and comparisons, we demonstrate the effectiveness of DiffusionGPT, showcasing its potential for pushing the boundaries of image synthesis in diverse domains.

翻译：扩散模型为图像生成领域开辟了新途径，导致开源平台上涌现出大量高质量模型。然而，当前文本到图像系统仍面临一个主要挑战——往往无法处理多样化输入，或局限于单一模型输出。现有统一化尝试通常集中在两个正交方向上：i) 在输入阶段解析多样化提示词；ii) 激活专家模型进行输出。为融合两者优势，我们提出DiffusionGPT，利用大语言模型（LLM）构建统一的生成系统，能够无缝适配各种类型的提示词并集成领域专家模型。DiffusionGPT基于先验知识为各类生成模型构建领域特定树。当输入内容时，LLM解析提示词并利用思维树引导选择合适模型，从而放松输入约束并确保跨领域的卓越性能。此外，我们引入优势数据库，通过人类反馈丰富思维树，使模型选择过程与人类偏好对齐。通过大量实验与对比，我们证明了DiffusionGPT的有效性，展示了其在突破图像合成领域边界方面的潜力。