A pivotal advancement in the progress of large language models (LLMs) is the emergence of the Mixture-of-Experts (MoE) LLMs. Compared to traditional LLMs, MoE LLMs can achieve higher performance with fewer parameters, but it is still hard to deploy them due to their immense parameter sizes. Different from previous weight pruning methods that rely on specifically designed hardware, this paper mainly aims to enhance the deployment efficiency of MoE LLMs by introducing plug-and-play expert-level sparsification techniques. Specifically, we propose, for the first time to our best knowledge, post-training approaches for task-agnostic and task-specific expert pruning and skipping of MoE LLMs, tailored to improve deployment efficiency while maintaining model performance across a wide range of tasks. Extensive experiments show that our proposed methods can simultaneously reduce model sizes and increase the inference speed, while maintaining satisfactory performance. Data and code will be available at https://github.com/Lucky-Lance/Expert_Sparsity.

翻译：大语言模型发展进程中的一个关键进步是混合专家大语言模型的出现。与传统大语言模型相比，混合专家模型能够以更少的参数实现更高的性能，但由于其庞大的参数量，实际部署仍然面临困难。不同于以往依赖专用硬件的权重剪枝方法，本文主要致力于通过引入即插即用的专家级稀疏化技术来提升混合专家大语言模型的部署效率。具体而言，据我们所知，我们首次提出了面向混合专家大语言模型的任务无关与任务特定专家剪枝及跳过的训练后处理方法，旨在提升部署效率的同时，保持模型在广泛任务上的性能表现。大量实验表明，我们提出的方法能够同时减小模型规模并提升推理速度，同时保持令人满意的性能。数据和代码将在 https://github.com/Lucky-Lance/Expert_Sparsity 公开。