Mixture-of-Experts (MoE) based large language models (LLMs), such as Qwen and DeepSeek, have recently emerged as an effective approach to improving model capacity without proportionally increasing computational cost. By replacing the conventional feed-forward network in dense LLMs with a set of experts and activating only a subset of them for each input token, MoE models significantly increase the total number of parameters while keeping the per-token computation relatively manageable. However, this dynamic and irregular expert activation pattern also introduces substantial expert loading overhead during inference, since the required experts must be fetched on demand according to token-dependent routing results. As a result, expert loading latency becomes a major source of performance and energy inefficiency. To this end, we first perform a comprehensive analysis of expert selection behavior in various MoE-based LLMs and applications, including language understanding and code generation. Our analysis reveals that, within each application domain, expert requests exhibit strong correlation across both adjacent MoE layers and consecutive decoding tokens, making future expert activations predictable. Based on this insight, we propose ST-MoE, a spatio-temporal expert prefetching framework that proactively stages experts ahead of use to overlap expert loading with ongoing computation. ST-MoE combines a lightweight runtime prediction mechanism that preserves the original routing behavior with a reconfigurable hardware design that efficiently supports dynamic expert prefetching. The combined effect of the prediction mechanism with the supporting hardware significantly improves MoE inference performance and energy efficiency while preserving model inference accuracy.

翻译：基于混合专家模型（MoE）的大语言模型（如Qwen和DeepSeek）近期作为一种提升模型容量而不成比例增加计算成本的有效方法脱颖而出。通过用一组专家取代传统密集大语言模型中的前馈网络，并为每个输入令牌仅激活其中的一部分，MoE模型显著增加了总参数量，同时使每令牌计算量保持在相对可控水平。然而，这种动态且不规则的专家激活模式在推理过程中也带来了显著的专家加载开销——因为所需专家必须根据令牌依赖的路由结果按需获取。由此，专家加载延迟成为性能和能效低下的主要来源。为此，我们首先针对多种基于MoE的大语言模型及其应用（包括语言理解和代码生成），全面分析了专家选择行为。分析表明，在各应用领域内，相邻MoE层之间及连续解码令牌之间的专家请求均呈现强相关性，使得未来专家激活具有可预测性。基于这一发现，我们提出ST-MoE——一种时空专家预取框架，该框架通过前瞻性预加载专家以将专家加载与进行中的计算重叠。ST-MoE结合了保持原始路由行为的轻量级运行时预测机制，以及高效支持动态专家预取的可重构硬件设计。预测机制与支持硬件的协同作用显著提升了MoE推理性能与能效，同时保持模型推理精度不变。