Large Language Model (LLM) training is frequently interrupted by a heterogeneous spectrum of failures, from common GPU crashes to catastrophic cluster-wide outages. Existing checkpointing systems rely on monolithic, single-tier storage backend, forcing a trade-off between state-saving overhead and recovery speed. We propose TierCheck, a cluster-aware tiered checkpointing system that aligns storage placement with failure heterogeneity. TierCheck adopts a three-tier design that maintains lightweight differential checkpoints in local and peer memory for fast localized recovery, while asynchronously migrating heavyweight base checkpoints to remote persistent storage. It also ensures strict global consistency across tiers without stalling training, and achieves fast cluster-aware checkpoint restoration during recovery. Evaluations on models up to 40 billion parameters show that TierCheck achieves low training overhead, reduces end-to-end checkpointing time to under 10s, and supports high-frequency checkpointing, ultimately striking an optimal balance between low-overhead persistence and fast recovery.

翻译：大规模语言模型（LLM）训练频繁受到异构故障的干扰，从常见的GPU崩溃到灾难性的集群级中断。现有检查点系统依赖单层单一存储后端，迫使在状态保存开销与恢复速度之间进行权衡。我们提出TierCheck，一种集群感知的分层检查点系统，将存储布局与故障异构性对齐。TierCheck采用三层架构：在本地和伙伴内存中维护轻量级差分检查点以实现快速局部恢复，同时异步将重量级基础检查点迁移至远程持久存储。该系统还能在不中断训练的情况下确保跨层严格全局一致性，并在恢复过程中实现快速集群感知的检查点还原。在高达400亿参数模型上的评估表明，TierCheck实现了低训练开销，将端到端检查点时间缩短至10秒以下，并支持高频检查点，最终在低开销持久性与快速恢复之间取得最优平衡。