The sparsely gated mixture of experts (MoE) architecture sends different inputs to different subnetworks, i.e., experts, through trainable routers. MoE reduces the training computation significantly for large models, but its deployment can be still memory or computation expensive for some downstream tasks. Model pruning is a popular approach to reduce inference computation, but its application in MoE architecture is largely unexplored. To the best of our knowledge, this paper provides the first provably efficient technique for pruning experts in finetuned MoE models. We theoretically prove that prioritizing the pruning of the experts with a smaller change of the routers l2 norm from the pretrained model guarantees the preservation of test accuracy, while significantly reducing the model size and the computational requirements. Although our theoretical analysis is centered on binary classification tasks on simplified MoE architecture, our expert pruning method is verified on large vision MoE models such as VMoE and E3MoE finetuned on benchmark datasets such as CIFAR10, CIFAR100, and ImageNet.

翻译：稀疏门控专家混合（MoE）架构通过可训练的路由器将不同输入分配给不同的子网络（即专家）。MoE显著降低了大型模型的训练计算量，但在某些下游任务中，其部署仍可能面临内存或计算成本过高的问题。模型剪枝是减少推理计算量的常用方法，但其在MoE架构中的应用尚未得到充分探索。据我们所知，本文首次提出了一种可证明高效的微调MoE模型专家剪枝技术。我们通过理论证明，优先剪枝那些路由器L2范数与预训练模型相比变化较小的专家，能够在保证测试精度的同时，显著降低模型规模和计算需求。尽管我们的理论分析主要针对简化MoE架构上的二分类任务，但所提出的专家剪枝方法在VMoE和E3MoE等大型视觉MoE模型上得到了验证，这些模型已在CIFAR10、CIFAR100和ImageNet等基准数据集上完成微调。