Selective parameter activation provided by Mixture-of-Expert (MoE) models have made them a popular choice in modern foundational models. However, MoEs face a fundamental tension when employed for serving. Batching, critical for performance in serving, forces the activation of all experts, thereby negating MoEs' benefits and exacerbating memory bandwidth bottlenecks. Existing work on efficient MoE inference are unable to resolve this tension even with extensive workload-specific tuning. We present LYNX, a system that enables efficient MoE inference in a workload-agnostic fashion. Exploiting several key observations that we uncover in this work, LYNX provides a light-weight run-time dynamic expert remapping technique that depends only on information already available in the models. Our evaluation of LYNX on four state-of-the-art model families across nine benchmarks shows that it achieves up to 1.23x improvement in throughput while simultaneously improving accuracy by up to 4% in the majority of the tasks, and incurs only a negligible accuracy loss of less than 1% points in significantly hard tasks. Further, LYNX is complementary to existing techniques where it additionally boosts their performance by up to 1.38x.

翻译：混合专家（Mixture-of-Expert, MoE）模型提供的选择性参数激活机制，使其成为现代基础模型的流行选择。然而，MoE模型在部署推理时面临一个根本性矛盾：批处理对推理性能至关重要，却会强制激活所有专家，这不仅抵消了MoE的优势，还加剧了内存带宽瓶颈。现有高效MoE推理方法即使经过大量面向特定工作负载的调优，仍无法解决这一矛盾。本文提出LYNX系统，以与工作负载无关的方式实现高效MoE推理。通过利用本工作中揭示的若干关键观察，LYNX提供了一种轻量级运行时动态专家重映射技术，该技术仅依赖模型中已有的信息。我们在九个基准测试中对四个先进模型系列评估LYNX，结果表明：其在多数任务中最高可实现1.23倍的吞吐量提升，同时准确率最高提升4%；在极困难任务中仅产生不足1个百分点的可忽略准确率损失。此外，LYNX与现有技术具有互补性，可进一步将其性能提升最高达1.38倍。