In recent years, advances in the large-scale pretraining of language and text-to-image models have revolutionized the field of machine learning. Yet, integrating these two modalities into a single, robust model capable of generating seamless multimodal outputs remains a significant challenge. To address this gap, we present the Joint Autoregressive Mixture (JAM) framework, a modular approach that systematically fuses existing text and image generation models. We also introduce a specialized, data-efficient instruction-tuning strategy, tailored for mixed-modal generation tasks. Our final instruct-tuned model demonstrates unparalleled performance in generating high-quality multimodal outputs and represents the first model explicitly designed for this purpose.

翻译：近年来，大规模语言模型与文生图模型的预训练技术革新了机器学习领域。然而，将这两种模态整合为单一、鲁棒的多模态输出生成模型仍面临重大挑战。为解决这一空白，我们提出联合自回归混合框架（JAM），这是一种模块化方法，可系统融合现有文本与图像生成模型。我们还引入了一种针对混合模态生成任务的数据高效指令调优策略。最终经指令调优的模型在生成高质量多模态输出方面展现出无与伦比的性能，并成为首个为这一目标明确设计的模型。