Training large language models (LLMs) efficiently while preserving model quality poses significant challenges, particularly with subbyte precision supported by state-of-the-art GPUs. Current mixed-precision training approaches either apply uniform precision to all GEMM operations or rely on heuristic-based methods that fail to generalize during training, leading to suboptimal convergence and instability. To address these challenges, this paper introduces SNIP, a fine-grained adaptive mixed-precision training framework for LLM pretraining that supports subbyte precision. SNIP periodically collects statistics on activations, gradients, and optimizer states to assess the precision loss impact on model quality. We define two key metrics: loss divergence in the forward pass, caused by quantization-induced increases in training loss, and weight divergence in the backward pass, which measures error propagation through gradients affecting model updates. These metrics guide an Integer Linear Programming (ILP) problem that systematically optimizes layerwise precision to minimize overall quality loss while meeting efficiency targets. Experiments on 1B, 3B, 7B and 70B Llama-like models demonstrate that SNIP consistently outperforms existing baselines, reducing FLOPs by up to 80% while preserving model quality across different model sizes and training phases with minimal computational overhead.

翻译：在大语言模型（LLM）训练中，如何在保持模型质量的同时实现高效训练，尤其是在当前先进GPU支持的亚字节精度下，面临着重大挑战。现有的混合精度训练方法要么对所有GEMM运算采用统一精度，要么依赖于启发式方法，这些方法在训练过程中泛化能力不足，导致收敛次优和不稳定。为应对这些挑战，本文提出了SNIP，一种面向LLM预训练的细粒度自适应混合精度训练框架，支持亚字节精度。SNIP定期收集激活值、梯度和优化器状态的统计信息，以评估精度损失对模型质量的影响。我们定义了两个关键指标：前向传播中的损失散度（由量化引起的训练损失增加所导致）和后向传播中的权重散度（衡量通过梯度传播并影响模型更新的误差）。这些指标指导一个整数线性规划（ILP）问题，该系统性地优化各层精度，在满足效率目标的同时最小化整体质量损失。在1B、3B、7B和70B规模的类Llama模型上的实验表明，SNIP始终优于现有基线方法，在保持不同模型规模和训练阶段模型质量的前提下，将FLOPs降低高达80%，且计算开销极小。