Mixture-of-experts (MoE) models that employ sparse activation have demonstrated effectiveness in significantly increasing the number of parameters while maintaining low computational requirements per token. However, recent studies have established that MoE models are inherently parameter-inefficient as the improvement in performance diminishes with an increasing number of experts. We hypothesize this parameter inefficiency is a result of all experts having equal capacity, which may not adequately meet the varying complexity requirements of different tokens or tasks, e.g., in a multilingual setting, languages based on their resource levels might require different capacities. In light of this, we propose Stratified Mixture of Experts(SMoE) models, which feature a stratified structure and can assign dynamic capacity to different tokens. We demonstrate the effectiveness of SMoE on two multilingual machine translation benchmarks, where it outperforms multiple state-of-the-art MoE models. On a diverse 15-language dataset, SMoE improves the translation quality over vanilla MoE by +0.93 BLEU points on average. Additionally, SMoE is parameter-efficient, matching vanilla MoE performance with around 50\% fewer parameters.

翻译：混合专家（MoE）模型通过采用稀疏激活机制，在显著增加参数量的同时维持每个令牌的低计算需求，已被证明具有有效性。然而，近期研究指出，MoE模型本质上是参数低效的，因为随着专家数量的增加，性能提升幅度逐渐减弱。我们假设这种参数低效源于所有专家具有相同容量，这可能导致无法充分满足不同令牌或任务（例如多语言场景下基于资源水平的需求不同的语言）的复杂度要求。基于此，我们提出分层混合专家（SMoE）模型，该模型具有分层结构，可为不同令牌分配动态容量。我们在两个多语言机器翻译基准上验证了SMoE的有效性，其表现优于多个最先进的MoE模型。在包含15种语言的多样化数据集上，SMoE相比原始MoE平均提升+0.93个BLEU分。此外，SMoE具有参数高效性，仅需约50%的参数即可达到与原始MoE相当的性能。