Mixture-of-Experts (MoE) models are increasingly used to serve LLMs at scale, but failures become common as deployment scale grows. Existing systems exhibit poor failure resilience: even a single worker failure triggers a coarse-grained, service-wide restart, discarding accumulated progress and halting the entire inference pipeline during recovery--an approach clearly ill-suited for latency-sensitive, LLM services. We present Tarragon, a resilient MoE inference framework that confines the failures impact to individual workers while allowing the rest of the pipeline to continue making forward progress. Tarragon exploits the natural separation between the attention and expert computation in MoE-based transformers, treating attention workers (AWs) and expert workers (EWs) as distinct failure domains. Tarragon introduces a reconfigurable datapath to mask failures by rerouting requests to healthy workers. On top of this datapath, Tarragon implements a self-healing mechanism that relaxes the tightly synchronized execution of existing MoE frameworks. For stateful AWs, Tarragon performs asynchronous, incremental KV cache checkpointing with per-request restoration, and for stateless EWs, it leverages residual GPU memory to deploy shadow experts. These together keep recovery cost and recomputation overhead extremely low. Our evaluation shows that, compared to state-of-the-art MegaScale-Infer, Tarragon reduces failure-induced stalls by 160-213x (from ~64 s down to 0.3-0.4 s) while preserving performance when no failures occur.

翻译：混合专家模型正日益广泛地用于大规模部署大语言模型，但随着部署规模的扩大，故障发生频率显著增加。现有系统表现出较差的故障恢复能力：即使单个工作节点发生故障，也会触发粗粒度的全局服务重启，导致已累积的计算进度被丢弃，并在恢复期间完全中断整个推理流水线——这种方法显然不适用于对延迟敏感的大语言模型服务。本文提出Tarragon，一种具备容错能力的MoE推理框架，该框架能将故障影响限制在单个工作节点内，同时允许流水线其余部分继续推进计算。Tarragon利用基于MoE的Transformer架构中注意力计算与专家计算天然分离的特性，将注意力工作节点与专家工作节点划分为独立的故障域。通过引入可重构数据通路，Tarragon能够通过将请求重定向至健康工作节点来屏蔽故障。在此数据通路基础上，Tarragon实现了自愈机制，该机制放松了现有MoE框架中严格同步的执行模式。对于有状态的注意力工作节点，Tarragon采用基于异步增量KV缓存检查点与逐请求恢复的机制；对于无状态的专家工作节点，则利用GPU剩余内存部署影子专家。这些设计共同将恢复成本与重计算开销降至极低水平。实验评估表明，相较于最先进的MegaScale-Infer系统，Tarragon将故障导致的停滞时间降低了160-213倍（从约64秒缩短至0.3-0.4秒），同时在无故障发生时仍能保持原有性能。