Mixture-of-Experts (MoE) models have recently demonstrated exceptional performance across a diverse range of applications. The principle of sparse activation in MoE models facilitates an offloading strategy, wherein active experts are maintained in GPU HBM, while inactive experts are stored in CPU DRAM. The efficacy of this approach, however, is fundamentally constrained by the limited bandwidth of the CPU-GPU interconnect. To mitigate this bottleneck, existing approaches have employed prefetching to accelerate MoE inference. These methods attempt to predict and prefetch the required experts using specially trained modules. Nevertheless, such techniques are often encumbered by significant training overhead and have shown diminished effectiveness on recent MoE models with fine-grained expert segmentation. In this paper, we propose MoBiLE, a plug-and-play offloading-based MoE inference framework with \textit{mixture of big-little experts}. It reduces the number of experts for unimportant tokens to half for acceleration while maintaining full experts for important tokens to guarantee model quality. Further, a dedicated fallback and prefetching mechanism is designed for switching between little and big experts to improve memory efficiency. We evaluate MoBiLE on four typical modern MoE architectures and challenging generative tasks. Our results show that MoBiLE achieves a speedup of 1.60x to 1.72x compared to the baseline on a consumer GPU system, with negligible degradation in accuracy.

翻译：混合专家（MoE）模型近期在多种应用中展现出卓越性能。MoE模型的稀疏激活特性支持一种卸载策略，即将活跃专家保留在GPU高带宽内存中，而非活跃专家存储在CPU内存中。然而，该方法的效果从根本上受限于CPU-GPU互连带宽的约束。为缓解这一瓶颈，现有方法采用预取技术来加速MoE推理。这些方法试图通过专门训练的模块预测并预取所需专家。但此类技术通常伴随显著的训练开销，且在采用细粒度专家分割的近期MoE模型上效果有限。本文提出MoBiLE，一种基于\textit{大-小专家混合}的即插即用式卸载MoE推理框架。该框架将非重要令牌的专家数量减半以实现加速，同时对重要令牌保留完整专家以保证模型质量。此外，我们设计了专用的回退与预取机制，用于实现小专家与大专家间的切换以提升内存效率。我们在四种典型现代MoE架构和具有挑战性的生成任务上评估MoBiLE。实验结果表明，在消费级GPU系统上，MoBiLE相比基线方法实现了1.60倍至1.72倍的加速，且精度损失可忽略不计。