Large language models (LLMs) have demonstrated impressive zero-shot abilities on a variety of open-ended tasks, while recent research has also explored the use of LLMs for multi-modal generation. In this study, we introduce mPLUG-Owl, a novel training paradigm that equips LLMs with multi-modal abilities through modularized learning of foundation LLM, a visual knowledge module, and a visual abstractor module. This approach can support multiple modalities and facilitate diverse unimodal and multimodal abilities through modality collaboration. The training paradigm of mPLUG-Owl involves a two-stage method for aligning image and text, which learns visual knowledge with the assistance of LLM while maintaining and even improving the generation abilities of LLM. In the first stage, the visual knowledge module and abstractor module are trained with a frozen LLM module to align the image and text. In the second stage, language-only and multi-modal supervised datasets are used to jointly fine-tune a low-rank adaption (LoRA) module on LLM and the abstractor module by freezing the visual knowledge module. We carefully build a visually-related instruction evaluation set OwlEval. Experimental results show that our model outperforms existing multi-modal models, demonstrating mPLUG-Owl's impressive instruction and visual understanding ability, multi-turn conversation ability, and knowledge reasoning ability. Besides, we observe some unexpected and exciting abilities such as multi-image correlation and scene text understanding, which makes it possible to leverage it for harder real scenarios, such as vision-only document comprehension. Our code, pre-trained model, instruction-tuned models, and evaluation set are available at https://github.com/X-PLUG/mPLUG-Owl. The online demo is available at https://www.modelscope.cn/studios/damo/mPLUG-Owl.

翻译：大型语言模型（LLMs）在多种开放任务上展现出惊人的零样本能力，近期研究也开始探索利用LLMs进行多模态生成。本研究提出mPLUG-Owl，一种新颖的训练范式，通过基础LLM、视觉知识模块和视觉抽象模块的模块化学习，赋予LLMs多模态能力。该方法支持多种模态，并通过模态协作促进多样的单模态与多模态能力。mPLUG-Owl的训练范式采用两阶段方法对齐图像与文本，借助LLM学习视觉知识，同时保持甚至提升LLM的生成能力。在第一阶段，视觉知识模块与抽象模块在冻结LLM模块的条件下进行训练，以对齐图像与文本。第二阶段，利用纯语言及多模态有监督数据集，在冻结视觉知识模块的同时，对LLM上的低秩适配（LoRA）模块和抽象模块进行联合微调。我们精心构建了视觉相关指令评估集OwlEval。实验结果表明，我们的模型优于现有多模态模型，展现了mPLUG-Owl出色的指令与视觉理解能力、多轮对话能力及知识推理能力。此外，我们还观察到一些意想不到且令人振奋的能力，如多图像关联和场景文本理解，这使得mPLUG-Owl有望应用于更复杂的实际场景，例如纯视觉文档理解。我们的代码、预训练模型、指令微调模型及评估集已发布在https://github.com/X-PLUG/mPLUG-Owl，在线演示可在https://www.modelscope.cn/studios/damo/mPLUG-Owl获取。