In large-scale LLM pre-training systems with 100k+ GPUs, failures become the norm rather than the exception, and restart costs can dominate wall-clock training time. However, existing fault-tolerance mechanisms are largely unprepared for this restart-dominant regime. To address this challenge, we propose SPARe - Stacked Parallelism with Adaptive Reordering - a fault-tolerance framework that masks node failures during gradient synchronization by stacking redundant data shards across parallelism groups and adaptively reordering execution. SPARe achieves availability comparable to traditional replication while maintaining near-constant computation overhead of only 2~3x, even under high redundancy where traditional replication would require linearly inflating overhead. We derive closed-form expressions for endurable failure count and computation overhead, validate them via SimGrid-based discrete-event simulation, and jointly optimize redundancy and checkpointing to minimize time-to-train. At extreme scale with up to 600k GPUs, SPARe reduces time-to-train by 40~50% compared to traditional replication.

翻译：在十万级以上GPU的大规模大语言模型预训练系统中，故障成为常态而非例外，重启开销可能主导实际训练时间。然而，现有的容错机制大多未针对这种重启主导的机制做好准备。为应对这一挑战，我们提出SPARe——堆叠并行与自适应重排序——一种通过在并行组间堆叠冗余数据分片并自适应重排序执行，从而在梯度同步期间屏蔽节点故障的容错框架。SPARe实现了与传统复制相当的可用性，同时即使在传统复制需要线性膨胀开销的高冗余场景下，仍能保持仅2~3倍的近乎恒定的计算开销。我们推导了可容忍故障数量与计算开销的闭式表达式，通过基于SimGrid的离散事件仿真进行验证，并联合优化冗余与检查点机制以最小化训练完成时间。在高达60万GPU的极端规模下，与传统复制相比，SPARe将训练完成时间减少了40~50%。