We present Chameleon, a family of early-fusion token-based mixed-modal models capable of understanding and generating images and text in any arbitrary sequence. We outline a stable training approach from inception, an alignment recipe, and an architectural parameterization tailored for the early-fusion, token-based, mixed-modal setting. The models are evaluated on a comprehensive range of tasks, including visual question answering, image captioning, text generation, image generation, and long-form mixed modal generation. Chameleon demonstrates broad and general capabilities, including state-of-the-art performance in image captioning tasks, outperforms Llama-2 in text-only tasks while being competitive with models such as Mixtral 8x7B and Gemini-Pro, and performs non-trivial image generation, all in a single model. It also matches or exceeds the performance of much larger models, including Gemini Pro and GPT-4V, according to human judgments on a new long-form mixed-modal generation evaluation, where either the prompt or outputs contain mixed sequences of both images and text. Chameleon marks a significant step forward in a unified modeling of full multimodal documents.

翻译：我们提出了Chameleon，一个基于早期融合的token化混合模态模型家族，能够理解和生成任意序列的图像与文本。我们概述了一种从初始阶段便稳定的训练方法、对齐方案以及专为早期融合、token化、混合模态场景定制的架构参数化设计。该模型在广泛的任务上进行了评估，包括视觉问答、图像描述、文本生成、图像生成以及长文本混合模态生成任务。Chameleon展现了广泛且通用的能力，在图像描述任务中取得了最先进的性能，在纯文本任务上优于Llama-2，同时与Mixtral 8x7B和Gemini-Pro等模型具有竞争力，并实现了非平凡级别的图像生成——所有这些均集成于单一模型中。根据一项新的长格式混合模态生成评估（其中提示或输出包含图像与文本的混合序列）的人类判断，Chameleon在性能上匹配或超越了包括Gemini Pro和GPT-4V在内的更大模型。Chameleon标志着在全模态文档统一建模方面迈出了重要一步。