Mixture-of-Experts (MoE) has emerged as a favorable architecture in the era of large models due to its inherent advantage, i.e., enlarging model capacity without incurring notable computational overhead. Yet, the realization of such benefits often results in ineffective GPU memory utilization, as large portions of the model parameters remain dormant during inference. Moreover, the memory demands of large models consistently outpace the memory capacity of contemporary GPUs. Addressing this, we introduce SiDA-MoE ($\textbf{S}$parsity-$\textbf{i}$nspired $\textbf{D}$ata-$\textbf{A}$ware), an efficient inference approach tailored for large MoE models. SiDA-MoE judiciously exploits both the system's main memory, which is now abundant and readily scalable, and GPU memory by capitalizing on the inherent sparsity on expert activation in MoE models. By adopting a data-aware perspective, SiDA-MoE achieves enhanced model efficiency with a neglectable performance drop. Specifically, SiDA-MoE attains a remarkable speedup in MoE inference with up to $3.93\times$ throughput increasing, up to $72\%$ latency reduction, and up to $80\%$ GPU memory saving with down to $1\%$ performance drop. This work paves the way for scalable and efficient deployment of large MoE models, even with constrained resources. Code is available at: https://github.com/timlee0212/SiDA-MoE.

翻译：混合专家模型（MoE）因其固有的优势（即在不引入显著计算开销的情况下扩展模型容量）已成为大模型时代的首选架构。然而，这种优势的实现在推理过程中往往导致GPU内存利用效率低下，因为模型参数的大部分在推理中处于休眠状态。此外，大模型的内存需求始终超过当代GPU的内存容量。针对这一问题，我们提出SiDA-MoE（$\textbf{S}$parsity-$\textbf{i}$nspired $\textbf{D}$ata-$\textbf{A}$ware），一种专为大规模MoE模型设计的高效推理方法。SiDA-MoE通过利用MoE模型中专家激活的固有稀疏性，巧妙结合当前充裕且易扩展的系统主内存与GPU内存。采用数据感知视角，SiDA-MoE在性能下降可忽略的情况下实现了模型效率的提升。具体而言，SiDA-MoE在MoE推理中取得了显著加速：吞吐量提升高达$3.93\times$、延迟降低高达$72\%$、GPU内存节省高达$80\%$，而性能下降仅约$1\%$。这项工作为在资源受限条件下大规模MoE模型的可扩展高效部署铺平了道路。代码获取地址：https://github.com/timlee0212/SiDA-MoE。