Mixture-of-Experts (MoE) models are increasingly used to serve LLMs at scale, but failures become common as deployment scale grows. Existing systems exhibit poor failure resilience: even a single worker failure triggers a coarse-grained, service-wide restart, discarding accumulated progress and halting the entire inference pipeline during recovery--an approach clearly ill-suited for latency-sensitive, LLM services. We present Tarragon, a resilient MoE inference framework that confines the failures impact to individual workers while allowing the rest of the pipeline to continue making forward progress. Tarragon exploits the natural separation between the attention and expert computation in MoE-based transformers, treating attention workers (AWs) and expert workers (EWs) as distinct failure domains. Tarragon introduces a reconfigurable datapath to mask failures by rerouting requests to healthy workers. On top of this datapath, Tarragon implements a self-healing mechanism that relaxes the tightly synchronized execution of existing MoE frameworks. For stateful AWs, Tarragon performs asynchronous, incremental KV cache checkpointing with per-request restoration, and for stateless EWs, it leverages residual GPU memory to deploy shadow experts. These together keep recovery cost and recomputation overhead extremely low. Our evaluation shows that, compared to state-of-the-art MegaScale-Infer, Tarragon reduces failure-induced stalls by 160-213x (from ~64 s down to 0.3-0.4 s) while preserving performance when no failures occur.

翻译：基于专家混合（Mixture-of-Experts, MoE）的模型正日益广泛地用于大规模部署大语言模型（LLM），但随着部署规模的扩大，故障发生变得频繁。现有系统展现出较差的故障弹性：即使是单个工作节点故障也会触发粗粒度的、服务范围的重启，丢弃已累积的计算进度，并在恢复期间中断整个推理流水线——这种方法显然不适合对延迟敏感的LLM服务。我们提出了Tarragon，一个弹性的MoE推理框架，它将故障影响限制在单个工作节点内，同时允许流水线的其余部分继续向前推进。Tarragon利用了基于MoE的Transformer中注意力计算与专家计算之间的天然分离，将注意力工作节点（AWs）和专家工作节点（EWs）视为独立的故障域。Tarragon引入了一种可重构的数据通路，通过将请求重路由至健康的工作节点来屏蔽故障。在此数据通路之上，Tarragon实现了一种自愈机制，该机制放松了现有MoE框架中紧密同步的执行方式。对于有状态的AWs，Tarragon执行异步的、增量的键值缓存检查点，并支持按请求进行恢复；对于无状态的EWs，它利用GPU的剩余内存部署影子专家。这些机制共同将恢复成本和重计算开销保持在极低水平。我们的评估表明，与最先进的MegaScale-Infer相比，Tarragon将故障导致的停顿减少了160-213倍（从约64秒降至0.3-0.4秒），同时在无故障发生时保持性能不变。