Fine-grained Mixture-of-Experts (MoE) models sparsely activate only a subset of experts per token, reducing activated computation while maintaining high model capacity. However, in memory-constrained inference scenarios, only a small set of experts can be cached. Experts not in the cache must be fetched from slow external storage (e.g., UFS), leading to frequent evictions and substantial I/O overhead. We propose ReMoE, a router fine-tuning framework designed to boost token-wise expert reuse. ReMoE biases the router toward recently selected experts, producing temporally stable routing that better matches cache locality constraints. By increasing short-horizon expert reuse, ReMoE reduces expert fetches from storage without adding inference-time computation. Experiments on DeepSeek and Qwen models show that ReMoE improves expert reuse by 26% while maintaining downstream task performance. Real-system evaluations further confirm these benefits, improving output throughput by 8.4% under vLLM GPU-CPU expert offloading and reducing TPOT by 43.6-49.8% under llama.cpp on Jetson Orin NX, corresponding to a 1.77-1.99$\times$ decode speedup across diverse workloads. Checkpoints and usage instructions are available at https://github.com/BUAA-OSCAR/ReMoE.

翻译：细粒度混合专家（MoE）模型仅对每个令牌稀疏激活少量专家，在保持高模型容量的同时减少了激活计算量。然而，在内存受限的推理场景中，仅能缓存少量专家。未缓存的专家必须从低速外部存储（如UFS）中获取，导致频繁的驱逐和显著的I/O开销。我们提出ReMoE，一个旨在提升令牌级专家复用的路由器微调框架。ReMoE使路由器偏向近期选中的专家，产生时间上稳定的路由，从而更好地匹配缓存局部性约束。通过增加短时间范围内的专家复用，ReMoE在不增加推理计算开销的情况下减少了从存储中获取专家的次数。在DeepSeek和Qwen模型上的实验表明，ReMoE在保持下游任务性能的同时将专家复用率提升了26%。真实系统评估进一步验证了这些优势：在vLLM GPU-CPU专家卸载场景下，输出吞吐量提升8.4%；在Jetson Orin NX上的llama.cpp场景中，TPOT降低43.6-49.8%，对应不同工作负载下1.77-1.99倍的解码加速。检查点和使用说明请访问https://github.com/BUAA-OSCAR/ReMoE。