In this paper, we introduce ILLUME, a unified multimodal large language model (MLLM) that seamlessly integrates multimodal understanding and generation capabilities within a single large language model through a unified next-token prediction formulation. To address the large dataset size typically required for image-text alignment, we propose to enhance data efficiency through the design of a vision tokenizer that incorporates semantic information and a progressive multi-stage training procedure. This approach reduces the dataset size to just 15M for pretraining -- over four times fewer than what is typically needed -- while achieving competitive or even superior performance with existing unified MLLMs, such as Janus. Additionally, to promote synergistic enhancement between understanding and generation capabilities, which is under-explored in previous works, we introduce a novel self-enhancing multimodal alignment scheme. This scheme supervises the MLLM to self-assess the consistency between text descriptions and self-generated images, facilitating the model to interpret images more accurately and avoid unrealistic and incorrect predictions caused by misalignment in image generation. Based on extensive experiments, our proposed ILLUME stands out and competes with state-of-the-art unified MLLMs and specialized models across various benchmarks for multimodal understanding, generation, and editing.

翻译：本文介绍了ILLUME，一个统一的多模态大语言模型，它通过统一的下一词预测框架，在一个单一的大语言模型中无缝集成了多模态理解与生成能力。针对图像-文本对齐通常需要大规模数据集的问题，我们提出通过设计一个融合语义信息的视觉分词器以及一个渐进式多阶段训练流程来提升数据效率。该方法将预训练所需的数据集规模减少至仅1500万——比通常所需数据量少四倍以上——同时在与现有统一多模态大语言模型（如Janus）的对比中，取得了相当甚至更优的性能。此外，为了促进理解与生成能力之间的协同增强（这一点在先前工作中探索不足），我们引入了一种新颖的自增强多模态对齐方案。该方案监督多模态大语言模型自我评估文本描述与自生成图像之间的一致性，从而帮助模型更准确地解读图像，并避免因图像生成中的错位而导致不现实和错误的预测。基于大量实验，我们提出的ILLUME表现突出，在各种多模态理解、生成和编辑的基准测试中，与最先进的统一多模态大语言模型及专用模型相竞争。