Large-scale LLM training is increasingly susceptible to hardware defects stemming from manufacturing escapes and silicon aging. These defects manifest as Silent Data Corruption (SDC) that perturb gradients and parameters throughout the training process. We present LLM-PRISM, a methodology to characterize LLM pre-training resilience to hardware faults. LLM-PRISM couples RTL-level GPU fault simulation with a stochastic injection engine embedded in Megatron-LM. Through 7,664 training runs across FP16, BF16, and FP8 regimes, we analyze how fault type, rate, and numeric format govern resilience. We find that while LLMs resist low-frequency faults, impact is highly non-uniform; critical datapaths and specific precision formats can induce catastrophic divergence even at moderate fault rates. This study provides the first hardware-grounded, pre-training characterization of SDC resilience.

翻译：大规模LLM训练日益受到制造缺陷和硅老化引发的硬件缺陷影响。这些缺陷表现为静默数据损坏，会持续扰动训练过程中的梯度与参数。本文提出LLM-PRISM方法体系，用于刻画LLM预训练对硬件故障的鲁棒性。该方法将RTL级GPU故障模拟与嵌入Megatron-LM的随机注入引擎相结合。通过FP16、BF16和FP8三种数值格式下的7,664次训练实验，我们分析了故障类型、故障率与数值格式对鲁棒性的影响规律。研究发现，虽然LLM对低频故障具有抵抗能力，但其影响呈现高度非均匀性；关键数据通路与特定精度格式即使在中度故障率下也可能引发灾难性发散。本研究首次提供了基于硬件的SDC鲁棒性预训练特征分析。