As distributed optimization scales to meet the demands of Large Language Model (LLM) training, hardware failures become increasingly non-negligible. Existing fault-tolerant training methods often introduce significant computational or memory overhead, demanding additional resources. To address this challenge, we propose Memory- and Computation-efficient Fault-tolerant Optimization (MeCeFO), a novel algorithm that ensures robust training with minimal overhead. When a computing node fails, MeCeFO seamlessly transfers its training task to a neighboring node while employing memory- and computation-efficient algorithmic optimizations to minimize the extra workload imposed on the neighboring node handling both tasks. MeCeFO leverages three key algorithmic designs: (i) Skip-connection, which drops the multi-head attention (MHA) module during backpropagation for memory- and computation-efficient approximation; (ii) Recomputation, which reduces activation memory in feedforward networks (FFNs); and (iii) Low-rank gradient approximation, enabling efficient estimation of FFN weight matrix gradients. Theoretically, MeCeFO matches the convergence rate of conventional distributed training, with a rate of $\mathcal{O}(1/\sqrt{nT})$, where n is the data parallelism size and T is the number of iterations. Empirically, MeCeFO maintains robust performance under high failure rates, incurring only a 4.18% drop in throughput, demonstrating 5.0$\times$ to 6.7$\times$ greater resilience than previous SOTA approaches. Codes are available at https://github.com/pkumelon/MeCeFO.

翻译：随着分布式优化规模的扩大以满足大语言模型（LLM）训练的需求，硬件故障变得日益不可忽视。现有的容错训练方法通常会引入显著的计算或内存开销，需要额外的资源。为应对这一挑战，我们提出了内存与计算高效的容错优化算法（MeCeFO），这是一种新颖的算法，能以最小开销确保鲁棒的训练。当计算节点发生故障时，MeCeFO会将其训练任务无缝转移至相邻节点，同时采用内存与计算高效的算法优化，以最小化相邻节点处理双重任务所承担的额外工作负载。MeCeFO利用了三个关键的算法设计：（i）跳跃连接，在反向传播过程中丢弃多头注意力（MHA）模块，以实现内存与计算高效的近似；（ii）重计算，以减少前馈网络（FFN）中的激活内存；（iii）低秩梯度近似，实现对FFN权重矩阵梯度的高效估计。理论上，MeCeFO的收敛速度与传统分布式训练相匹配，收敛速率为$\mathcal{O}(1/\sqrt{nT})$，其中n为数据并行规模，T为迭代次数。实证结果表明，MeCeFO在高故障率下仍能保持鲁棒性能，吞吐量仅下降4.18%，其容错能力比先前的最先进（SOTA）方法高出5.0倍至6.7倍。代码可在https://github.com/pkumelon/MeCeFO获取。