To help the open-source community have a better understanding of Mixture-of-Experts (MoE) based large language models (LLMs), we train and release OpenMoE, a series of fully open-sourced and reproducible decoder-only MoE LLMs, ranging from 650M to 34B parameters and trained on up to over 1T tokens. Our investigation confirms that MoE-based LLMs can offer a more favorable cost-effectiveness trade-off than dense LLMs, highlighting the potential effectiveness for future LLM development. One more important contribution of this study is an in-depth analysis of the routing mechanisms within our OpenMoE models, leading to three significant findings: Context-Independent Specialization, Early Routing Learning, and Drop-towards-the-End. We discovered that routing decisions in MoE models are predominantly based on token IDs, with minimal context relevance. The token-to-expert assignments are determined early in the pre-training phase and remain largely unchanged. This imperfect routing can result in performance degradation, particularly in sequential tasks like multi-turn conversations, where tokens appearing later in a sequence are more likely to be dropped. Finally, we rethink our design based on the above-mentioned observations and analysis. To facilitate future MoE LLM development, we propose potential strategies for mitigating the issues we found and further improving off-the-shelf MoE LLM designs.

翻译：为帮助开源社区更深入理解基于混合专家（MoE）的大型语言模型（LLM），我们训练并发布了OpenMoE系列——一系列完全开源且可复现的纯解码器MoE大语言模型，参数规模从6.5亿到340亿，训练数据量超过1万亿令牌。研究表明，基于MoE的大语言模型相比密集模型能实现更优的成本效益权衡，凸显其对未来LLM发展的潜在有效性。本研究另一重要贡献是对OpenMoE模型路由机制的深度剖析，得出三项关键发现：上下文无关的专家专业化、路由学习的早期固化，以及序列末尾的专家丢弃。我们观察到，MoE模型的路由决策主要基于令牌ID，与上下文关联甚微。令牌与专家的分配在预训练早期阶段便已确定，且后期变化极小。这种非完美路由会导致性能下降，尤其在多轮对话等序列任务中，序列尾部令牌更易被丢弃。最后，基于上述观察与分析，我们重新审视模型设计，并提出缓解所发现问题的潜在策略，以进一步改进现有MoE大语言模型架构。