Large language models, such as OpenAI's ChatGPT, have demonstrated exceptional language understanding capabilities in various NLP tasks. Sparsely activated mixture-of-experts (MoE) has emerged as a promising solution for scaling models while maintaining a constant number of computational operations. Existing MoE model adopts a fixed gating network where each token is computed by the same number of experts. However, this approach contradicts our intuition that the tokens in each sequence vary in terms of their linguistic complexity and, consequently, require different computational costs. Little is discussed in prior research on the trade-off between computation per token and model performance. This paper introduces adaptive gating in MoE, a flexible training strategy that allows tokens to be processed by a variable number of experts based on expert probability distribution. The proposed framework preserves sparsity while improving training efficiency. Additionally, curriculum learning is leveraged to further reduce training time. Extensive experiments on diverse NLP tasks show that adaptive gating reduces at most 22.5% training time while maintaining inference quality. Moreover, we conduct a comprehensive analysis of the routing decisions and present our insights when adaptive gating is used.

翻译：大型语言模型（如OpenAI的ChatGPT）已在各类自然语言处理任务中展现出卓越的语言理解能力。稀疏激活的混合专家模型（MoE）作为一种在保持恒定计算量的同时扩展模型规模的解决方案而备受关注。现有MoE模型采用固定门控网络架构，每个令牌均由相同数量的专家计算处理。然而，这种方法与我们的直觉相悖——每个序列中的令牌在语言复杂性上存在差异，因此需要不同的计算成本。以往研究极少探讨单令牌计算量与模型性能之间的权衡问题。本文提出了一种基于MoE的自适应门控机制，这是一种灵活的训练策略，允许令牌根据专家概率分布由可变数量的专家进行处理。所提出的框架在保持稀疏性的同时提升了训练效率。此外，利用课程学习策略进一步缩短训练时间。在多种NLP任务上的广泛实验表明，自适应门控机制在保持推理质量的同时，最多可减少22.5%的训练时间。本文还对路由决策进行了全面分析，并阐述了使用自适应门控时的相关见解。