Mixture-of-Experts (MoE) models facilitate edge deployment by decoupling model capacity from active computation, yet their large memory footprint drives the need for GPU systems with near-data processing (NDP) capabilities that offload experts to dedicated processing units. However, deploying MoE models on such edge-based GPU-NDP systems faces three critical challenges: 1) severe load imbalance across NDP units due to non-uniform expert selection and expert parallelism, 2) insufficient GPU utilization during expert computation within NDP units, and 3) extensive data pre-profiling necessitated by unpredictable expert activation patterns for pre-fetching. To address these challenges, this paper proposes an efficient inference framework featuring three key optimizations. First, the underexplored tensor parallelism in MoE inference is exploited to partition and compute large expert parameters across multiple NDP units simultaneously towards edge low-batch scenarios. Second, a load-balancing-aware scheduling algorithm distributes expert computations across NDP units and GPU to maximize resource utilization. Third, a dataset-free pre-fetching strategy proactively loads frequently accessed experts to minimize activation delays. Experimental results show that our framework enables GPU-NDP systems to achieve 2.41x on average and up to 2.56x speedup in end-to-end latency compared to state-of-the-art approaches, significantly enhancing MoE inference efficiency in resource-constrained environments.

翻译：混合专家（Mixture-of-Experts, MoE）模型通过解耦模型容量与激活计算，促进了边缘部署，但其庞大的内存占用推动了对具备近数据处理（near-data processing, NDP）能力的GPU系统的需求，这类系统可将专家卸载至专用处理单元。然而，在此类基于边缘的GPU-NDP系统上部署MoE模型面临三个关键挑战：1) 由于非均匀的专家选择与专家并行，导致NDP单元间严重的负载不均衡；2) NDP单元内进行专家计算时GPU利用率不足；3) 为预取数据需要进行大量数据预分析，这源于难以预测的专家激活模式。为应对这些挑战，本文提出一种高效的推理框架，包含三项关键优化。首先，利用MoE推理中尚未充分探索的张量并行性，在边缘低批次场景下，将大型专家参数分区并跨多个NDP单元同时计算。其次，一种负载均衡感知的调度算法将专家计算分配至NDP单元与GPU，以最大化资源利用率。第三，一种无需数据集的预取策略主动加载频繁访问的专家，以最小化激活延迟。实验结果表明，与最先进方法相比，我们的框架使GPU-NDP系统在端到端延迟上平均实现2.41倍、最高达2.56倍的加速，显著提升了资源受限环境下的MoE推理效率。