In this paper, we introduce a novel dynamic expert selection framework for Mixture of Experts (MoE) models, aiming to enhance computational efficiency and model performance by adjusting the number of activated experts based on input difficulty. Unlike traditional MoE approaches that rely on fixed Top-K routing, which activates a predetermined number of experts regardless of the input's complexity, our method dynamically selects experts based on the confidence level in expert selection for each input. This allows for a more efficient utilization of computational resources, activating more experts for complex tasks requiring advanced reasoning and fewer for simpler tasks. Through extensive evaluations, our dynamic routing method demonstrates substantial improvements over conventional Top-2 routing across various benchmarks, achieving an average improvement of 0.7% with less than 90% activated parameters. Further analysis shows our model dispatches more experts to tasks requiring complex reasoning skills, like BBH, confirming its ability to dynamically allocate computational resources in alignment with the input's complexity. Our findings also highlight a variation in the number of experts needed across different layers of the transformer model, offering insights into the potential for designing heterogeneous MoE frameworks. The code and models are available at https://github.com/ZhenweiAn/Dynamic_MoE.

翻译：在本文中，我们提出了一种新颖的混合专家（MoE）模型动态专家选择框架，旨在根据输入难度调整激活专家数量，以提升计算效率和模型性能。传统MoE方法依赖固定的Top-K路由机制，无论输入复杂度如何，都会激活预定数量的专家。与此不同，我们的方法基于每个输入的专家选择置信度动态选择专家。这使得计算资源得到更高效的利用：对于需要高级推理的复杂任务，激活更多专家；对于简单任务，则激活较少专家。通过广泛评估，我们的动态路由方法在多个基准测试中相较于传统Top-2路由取得了显著改进，平均性能提升0.7%，且激活参数少于90%。进一步分析表明，我们的模型在处理需要复杂推理能力的任务（如BBH）时分配了更多专家，证实其能够根据输入复杂度动态分配计算资源。研究结果还揭示了Transformer模型不同层所需专家数量的差异，这为设计异构MoE框架提供了潜在思路。代码和模型已开源在https://github.com/ZhenweiAn/Dynamic_MoE。