Large Vision-Language Models (LVLMs) have demonstrated remarkable performance across multi-modal tasks by scaling model size and training data. However, these dense LVLMs incur significant computational costs and motivate the exploration of sparse Mixture of Experts (MoE) architectures. While MoE improve parameter efficiency, effectively applying MoE to simultaneously model modality-specific features and cross-modal associations in LVLMs remains challenging. In this work, we propose to incorporate Mixture of Intra- and Inter-Modality Experts (MoIIE) to LVLMs. For each token, expert routing is guided by its modality, directing tokens to their respective intra-modality experts as well as a shared pool of inter-modality experts, enabling the model to jointly learn rich intra-modal features and cross-modal interactions. We further introduce an effective and straightforward two-stage training strategy, which facilitates the direct activation of both MoE and multi-modal capabilities. Extensive experiments across different data scales and LLM backbone demonstrate the effectiveness, efficiency and generality of our approach. Notably, our MoIIE models with 5.5B and 11.3B activated parameters match or even surpass the performance of existing advanced open-source MoE-LLMs based multi-modal models that involve more activated parameters. The code is available at https://github.com/AlenjandroWang/MoIIE.

翻译：大规模视觉语言模型（LVLMs）通过扩展模型规模与训练数据，在多模态任务中展现出卓越性能。然而，这些密集型LVLMs需要高昂的计算成本，这促使研究者探索稀疏的专家混合（MoE）架构。尽管MoE提升了参数效率，但如何将其有效应用于LVLMs中以同时建模模态特定特征与跨模态关联仍具挑战性。本研究提出在LVLMs中引入模态内与模态间专家混合（MoIIE）机制。针对每个输入标记，专家路由由其模态属性引导，将其分配至相应的模态内专家池及共享的模态间专家池，使模型能够协同学习丰富的模态内特征与跨模态交互。我们进一步提出一种高效简洁的两阶段训练策略，直接激活模型的MoE能力与多模态能力。在不同数据规模与大语言模型骨干网络上的大量实验验证了本方法的有效性、高效性与普适性。值得注意的是，我们激活参数量为55亿与113亿的MoIIE模型，在性能上达到甚至超越了现有激活参数量更大的先进开源MoE-LLMs多模态模型。代码已发布于https://github.com/AlenjandroWang/MoIIE。