Recently, growing interest has been aroused in extending the multimodal capability of large language models (LLMs), e.g., vision-language (VL) learning, which is regarded as the next milestone of artificial general intelligence. However, existing solutions are prohibitively expensive, which not only need to optimize excessive parameters, but also require another large-scale pre-training before VL instruction tuning. In this paper, we propose a novel and affordable solution for the effective VL adaption of LLMs, called Mixture-of-Modality Adaptation (MMA). Instead of using large neural networks to connect the image encoder and LLM, MMA adopts lightweight modules, i.e., adapters, to bridge the gap between LLMs and VL tasks, which also enables the joint optimization of the image and language models. Meanwhile, MMA is also equipped with a routing algorithm to help LLMs achieve an automatic shift between single- and multi-modal instructions without compromising their ability of natural language understanding. To validate MMA, we apply it to a recent LLM called LLaMA and term this formed large vision-language instructed model as LaVIN. To validate MMA and LaVIN, we conduct extensive experiments under two setups, namely multimodal science question answering and multimodal dialogue. The experimental results not only demonstrate the competitive performance and the superior training efficiency of LaVIN than existing multimodal LLMs, but also confirm its great potential as a general-purpose chatbot. More importantly, the actual expenditure of LaVIN is extremely cheap, e.g., only 1.4 training hours with 3.8M trainable parameters, greatly confirming the effectiveness of MMA. Our project is released at https://luogen1996.github.io/lavin.

翻译：近年来，扩展大语言模型（LLMs）多模态能力（如视觉语言（VL）学习）的研究日益引起关注，这被视为人工通用智能的下一个里程碑。然而，现有解决方案成本高昂，不仅需要优化过量参数，还需在VL指令微调前进行大规模预训练。本文提出一种新颖且经济可行的LLMs视觉语言适配方法，称为模态混合适配（MMA）。MMA无需使用大型神经网络连接图像编码器与LLM，而是采用轻量级模块（即适配器）弥合LLMs与VL任务之间的鸿沟，同时实现图像与语言模型的联合优化。此外，MMA配备路由算法，帮助LLMs在单模态与多模态指令间自动切换，且不损害其自然语言理解能力。为验证MMA，我们将其应用于近期发布的LLaMA模型，并将由此形成的大规模视觉语言指令微调模型命名为LaVIN。我们在多模态科学问答与多模态对话两种场景下开展大量实验。实验结果不仅证明LaVIN与现有多模态LLMs相比具有竞争性性能与卓越训练效率，还确认其作为通用型聊天机器人的巨大潜力。更重要的是，LaVIN的实际开销极为廉价——仅需1.4小时训练时长与380万可训练参数，充分验证了MMA的有效性。项目已发布于https://luogen1996.github.io/lavin。