In large-scale LLM pre-training systems with 100k+ GPUs, failures become the norm rather than the exception, and restart costs can dominate wall-clock training time. However, existing fault-tolerance mechanisms are largely unprepared for this restart-dominant regime. To address this challenge, we propose SPARe - Stacked Parallelism with Adaptive Reordering - a fault-tolerance framework that masks node failures during gradient synchronization by stacking redundant data shards across parallelism groups and adaptively reordering execution. SPARe achieves availability comparable to traditional replication while maintaining near-constant computation overhead of only 2~3x, even under high redundancy where traditional replication would require linearly inflating overhead. We derive closed-form expressions for endurable failure count and computation overhead, validate them via SimGrid-based discrete-event simulation, and jointly optimize redundancy and checkpointing to minimize time-to-train. At extreme scale with up to 600k GPUs, SPARe reduces time-to-train by 40~50% compared to traditional replication.

翻译：在配备10万+GPU的大规模LLM预训练系统中，故障已成为常态而非例外，重启开销可能主导墙上时钟训练时间。然而，现有容错机制尚未充分应对这种以重启为主的运行模式。针对这一挑战，我们提出SPARe——一种通过跨并行组堆叠冗余数据分片并自适应重排序执行流程，在梯度同步期间掩盖节点故障的容错框架。SPARe在维持仅2~3倍近恒定计算开销的同时，实现了与传统复制方案相当的可用性，即使在高冗余场景下传统复制需线性增加开销亦如此。我们推导了可容忍故障数量与计算开销的闭式表达式，通过基于SimGrid的离散事件仿真进行验证，并联合优化冗余与检查点设置以最小化训练总时长。在多达60万GPU的极大规模部署中，相比传统复制方案，SPARe可将训练总时长降低40%~50%。