Recent advancements in Multimodal Large Language Models (LLMs) have focused primarily on scaling by increasing text-image pair data and enhancing LLMs to improve performance on multimodal tasks. However, these scaling approaches are computationally expensive and overlook the significance of improving model capabilities from the vision side. Inspired by the successful applications of Mixture-of-Experts (MoE) in LLMs, which improves model scalability during training while keeping inference costs similar to those of smaller models, we propose CuMo. CuMo incorporates Co-upcycled Top-K sparsely-gated Mixture-of-experts blocks into both the vision encoder and the MLP connector, thereby enhancing the multimodal LLMs with minimal additional activated parameters during inference. CuMo first pre-trains the MLP blocks and then initializes each expert in the MoE block from the pre-trained MLP block during the visual instruction tuning stage. Auxiliary losses are used to ensure a balanced loading of experts. CuMo outperforms state-of-the-art multimodal LLMs across various VQA and visual-instruction-following benchmarks using models within each model size group, all while training exclusively on open-sourced datasets. The code and model weights for CuMo are open-sourced at https://github.com/SHI-Labs/CuMo.

翻译：近期多模态大语言模型（LLMs）的进展主要聚焦于通过增加文本-图像配对数据量和增强LLMs来提升多模态任务性能。然而，此类扩展方法计算成本高昂，且忽视了从视觉维度提升模型能力的重要性。受混合专家（MoE）在LLMs中成功应用的启发——该方法在训练阶段提升模型可扩展性，同时保持与较小模型相近的推理成本——我们提出CuMo。CuMo将协同升级的Top-K稀疏门控混合专家模块引入视觉编码器和MLP连接器，从而在推理期间以极少的额外激活参数增强多模态LLMs。该方法首先预训练MLP模块，然后在视觉指令微调阶段，将MoE模块中的每个专家初始化为预训练MLP模块的副本。通过辅助损失函数确保专家负载均衡。在各类模型大小分组中，CuMo在多个VQA和视觉指令遵循基准测试上均超越现有最优多模态LLMs，且所有模型仅使用开源数据集进行训练。CuMo的代码与模型权重已开源至https://github.com/SHI-Labs/CuMo。