In this work, we study the problem of generating novel images from complex multimodal prompt sequences. While existing methods achieve promising results for text-to-image generation, they often struggle to capture fine-grained details from lengthy prompts and maintain contextual coherence within prompt sequences. Moreover, they often result in misaligned image generation for prompt sequences featuring multiple objects. To address this, we propose a Multi-modal Generation via Cross-Modal In-Context Learning (MGCC) method that generates novel images from complex multimodal prompt sequences by leveraging the combined capabilities of large language models (LLMs) and diffusion models. Our MGCC comprises a novel Cross-Modal Refinement module to explicitly learn cross-modal dependencies between the text and image in the LLM embedding space, and a contextual object grounding module to generate object bounding boxes specifically targeting scenes with multiple objects. Our MGCC demonstrates a diverse range of multimodal capabilities, like novel image generation, the facilitation of multimodal dialogue, and generation of texts. Experimental evaluations on two benchmark datasets, demonstrate the effectiveness of our method. On Visual Story Generation (VIST) dataset with multimodal inputs, our MGCC achieves a CLIP Similarity score of $0.652$ compared to SOTA GILL $0.641$. Similarly, on Visual Dialogue Context (VisDial) having lengthy dialogue sequences, our MGCC achieves an impressive CLIP score of $0.660$, largely outperforming existing SOTA method scoring $0.645$. Code: https://github.com/VIROBO-15/MGCC

翻译：本研究探讨了从复杂多模态提示序列生成新颖图像的问题。尽管现有方法在文本到图像生成方面取得了有希望的结果，但它们往往难以从冗长提示中捕捉细粒度细节，并在提示序列内保持上下文连贯性。此外，对于包含多个对象的提示序列，它们经常导致图像生成错位。为解决这些问题，我们提出了一种基于跨模态上下文学习的多模态生成方法，通过利用大语言模型和扩散模型的组合能力，从复杂多模态提示序列生成新颖图像。我们的MGCC包含一个新颖的跨模态细化模块，用于在LLM嵌入空间中显式学习文本与图像之间的跨模态依赖关系，以及一个上下文对象定位模块，专门针对多对象场景生成对象边界框。我们的MGCC展示了多样化的多模态能力，如新颖图像生成、促进多模态对话和文本生成。在两个基准数据集上的实验评估证明了我们方法的有效性。在具有多模态输入的视觉故事生成数据集上，我们的MGCC获得了0.652的CLIP相似度分数，优于当前最优方法GILL的0.641。同样，在包含冗长对话序列的视觉对话上下文数据集上，我们的MGCC取得了0.660的出色CLIP分数，大幅超越了现有最优方法的0.645。代码：https://github.com/VIROBO-15/MGCC