As large language models scale, training them requires thousands of GPUs over extended durations--making frequent failures an inevitable reality. While checkpointing remains the primary fault-tolerance mechanism, existing methods fall short when applied to Mixture-of-Experts (MoE) models. Due to their substantially larger training state, MoE models exacerbate checkpointing overheads, often causing costly stalls or prolonged recovery that severely degrade training efficiency. We present MoEvement, a distributed, in-memory checkpointing system tailored for MoE models. MoEvement is built on three key ideas: (1) sparse checkpointing, which incrementally snapshots subsets of experts across iterations to reduce overhead; (2) a sparse-to-dense checkpoint conversion mechanism that incrementally reconstructs consistent dense checkpoints from sparse snapshots; and (3) upstream logging of activations and gradients at pipeline-stage boundaries, enabling localized recovery without re-executing unaffected workers. Evaluations across diverse MoE models with up to 64 experts show that MoEvement reduces checkpointing overhead by up to $4\times$ and recovery overhead by up to $31\times$ compared to state-of-the-art approaches, sustaining ETTR $\ge 0.94$ even under frequent failures (MTBF as low as 10 minutes) and delivering up to $8\times$ overall training speedup, all without compromising synchronous training semantics. Overall, MoEvement offers a robust and scalable fault-tolerance solution for the next generation of sparsely activated models.

翻译：随着大语言模型规模不断扩展，其训练过程需数千GPU长时间运行，频繁故障成为必然挑战。尽管检查点机制仍是主要容错手段，但现有方法应用于混合专家（MoE）模型时存在明显不足。由于训练状态大幅膨胀，MoE模型加剧了检查点开销，常引发代价高昂的训练停滞或长时间恢复，严重降低训练效率。我们提出MoEvement——面向MoE模型的分布式内存检查点系统。该系统基于三个核心理念：（1）稀疏检查点，通过跨迭代增量式快照专家子集降低开销；（2）稀疏到稠密的检查点转换机制，从稀疏快照逐步重建一致性稠密检查点；（3）在流水线阶段边界对激活值与梯度实施上游日志记录，实现局部恢复而无需重放未受影响的工作节点。在包含多达64位专家的多种MoE模型上的评估表明，相比当前最先进方法，MoEvement将检查点开销降低最高4倍，恢复开销降低最高31倍，即使在高频故障场景（最低平均故障间隔时间10分钟）下仍能保持ETTR≥0.94，同时实现整体训练速度提升达8倍，且不破坏同步训练语义。总体而言，MoEvement为下一代稀疏激活模型提供了稳健可扩展的容错解决方案。