Mixture-of-Experts (MoE) large language models (LLM) have memory requirements that often exceed the GPU memory capacity, requiring costly parameter movement from secondary memories to the GPU for expert computation. In this work, we present Mixture of Near-Data Experts (MoNDE), a near-data computing solution that efficiently enables MoE LLM inference. MoNDE reduces the volume of MoE parameter movement by transferring only the $\textit{hot}$ experts to the GPU, while computing the remaining $\textit{cold}$ experts inside the host memory device. By replacing the transfers of massive expert parameters with the ones of small activations, MoNDE enables far more communication-efficient MoE inference, thereby resulting in substantial speedups over the existing parameter offloading frameworks for both encoder and decoder operations.

翻译：专家混合（MoE）大型语言模型（LLM）的内存需求常超出GPU内存容量，需要将参数从辅助存储器频繁迁移至GPU以执行专家计算，代价高昂。本文提出近数据专家混合（MoNDE），一种近数据计算解决方案，可高效支持MoE LLM推理。MoNDE通过仅将$\textit{热}$专家迁移至GPU，同时在主机内存设备内计算其余$\textit{冷}$专家，显著减少了MoE参数的迁移量。通过以小型激活值的传输替代海量专家参数的迁移，MoNDE实现了通信效率更高的MoE推理，从而在编码器与解码器运算中均较现有参数卸载框架获得显著加速。