Text-to-image (T2I) generation models have significantly advanced in recent years. However, effective interaction with these models is challenging for average users due to the need for specialized prompt engineering knowledge and the inability to perform multi-turn image generation, hindering a dynamic and iterative creation process. Recent attempts have tried to equip Multi-modal Large Language Models (MLLMs) with T2I models to bring the user's natural language instructions into reality. Hence, the output modality of MLLMs is extended, and the multi-turn generation quality of T2I models is enhanced thanks to the strong multi-modal comprehension ability of MLLMs. However, many of these works face challenges in identifying correct output modalities and generating coherent images accordingly as the number of output modalities increases and the conversations go deeper. Therefore, we propose DialogGen, an effective pipeline to align off-the-shelf MLLMs and T2I models to build a Multi-modal Interactive Dialogue System (MIDS) for multi-turn Text-to-Image generation. It is composed of drawing prompt alignment, careful training data curation, and error correction. Moreover, as the field of MIDS flourishes, comprehensive benchmarks are urgently needed to evaluate MIDS fairly in terms of output modality correctness and multi-modal output coherence. To address this issue, we introduce the Multi-modal Dialogue Benchmark (DialogBen), a comprehensive bilingual benchmark designed to assess the ability of MLLMs to generate accurate and coherent multi-modal content that supports image editing. It contains two evaluation metrics to measure the model's ability to switch modalities and the coherence of the output images. Our extensive experiments on DialogBen and user study demonstrate the effectiveness of DialogGen compared with other State-of-the-Art models.

翻译：文本到图像（T2I）生成模型近年来取得了显著进展。然而，普通用户与这些模型进行有效交互仍面临挑战，这主要源于需要专门的提示工程知识以及无法执行多轮图像生成，从而阻碍了动态迭代的创作过程。近期的一些尝试试图为多模态大语言模型（MLLMs）配备T2I模型，以将用户的自然语言指令转化为现实。因此，MLLMs的输出模态得以扩展，并且得益于MLLMs强大的多模态理解能力，T2I模型的多轮生成质量也得到了提升。然而，随着输出模态数量的增加和对话的深入，许多此类工作面临着识别正确输出模态并据此生成连贯图像的挑战。为此，我们提出了DialogGen，这是一个有效的流程，用于对齐现成的MLLMs和T2I模型，以构建一个用于多轮文本到图像生成的多模态交互式对话系统（MIDS）。它由绘图提示对齐、精心的训练数据整理和错误纠正组成。此外，随着MIDS领域的蓬勃发展，迫切需要全面的基准来公平评估MIDS在输出模态正确性和多模态输出连贯性方面的表现。为解决这一问题，我们引入了多模态对话基准（DialogBen），这是一个全面的双语基准，旨在评估MLLMs生成准确、连贯且支持图像编辑的多模态内容的能力。它包含两个评估指标，用于衡量模型切换模态的能力以及输出图像的连贯性。我们在DialogBen上的大量实验和用户研究表明，与其他最先进的模型相比，DialogGen具有显著的有效性。