For Large Vision-Language Models (LVLMs), scaling the model can effectively improve performance. However, expanding model parameters significantly increases the training and inferring costs, as all model parameters are activated for each token in the calculation. In this work, we propose a novel training strategy MoE-tuning for LVLMs, which can constructing a sparse model with an outrageous number of parameter but a constant computational cost, and effectively addresses the performance degradation typically associated with multi-modal learning and model sparsity. Furthermore, we present the MoE-LLaVA framework, a MoE-based sparse LVLM architecture. This framework uniquely activates only the top-k experts through routers during deployment, keeping the remaining experts inactive. Our extensive experiments highlight the excellent capabilities of MoE-LLaVA in visual understanding and its potential to reduce hallucinations in model outputs. Remarkably, with just 3 billion sparsely activated parameters, MoE-LLaVA demonstrates performance comparable to the LLaVA-1.5-7B on various visual understanding datasets and even surpasses the LLaVA-1.5-13B in object hallucination benchmarks. Through MoE-LLaVA, we aim to establish a baseline for sparse LVLMs and provide valuable insights for future research in developing more efficient and effective multi-modal learning systems. Code is released at \url{https://github.com/PKU-YuanGroup/MoE-LLaVA}.

翻译：对于大型视觉-语言模型（LVLMs），扩展模型规模可有效提升性能。然而，增加模型参数会显著提高训练与推理成本，因为每个token在计算中需激活全部模型参数。本文提出一种针对LVLMs的新型训练策略——MoE-tuning，该策略能构建参数规模庞大但计算成本恒定的稀疏模型，并有效解决多模态学习与模型稀疏性通常伴随的性能退化问题。此外，我们提出MoE-LLaVA框架，这是一种基于MoE的稀疏LVLM架构。该框架在部署时通过路由器仅激活前k个专家单元，其余专家保持非激活状态。大量实验表明，MoE-LLaVA在视觉理解方面具有卓越能力，并具备减少模型输出幻觉的潜力。值得关注的是，仅凭30亿稀疏激活参数，MoE-LLaVA即在多项视觉理解数据集上展现与LLaVA-1.5-7B相当的性能，并在物体幻觉基准测试中超越LLaVA-1.5-13B。通过MoE-LLaVA，我们旨在为稀疏LVLM建立基线，并为未来更高效的多模态学习系统研究提供重要启示。代码已开源至\url{https://github.com/PKU-YuanGroup/MoE-LLaVA}。