The NVFP4 lower-precision format, supported in hardware by NVIDIA Blackwell GPUs, promises to allow, for the first time, end-to-end fully-quantized pre-training of massive models such as LLMs. Yet, existing quantized training methods still sacrifice some of the representation capacity of this format in favor of more accurate unbiased quantized gradient estimation by stochastic rounding (SR), losing noticeable accuracy relative to standard FP16 and FP8 training. In this paper, improve the state of the art for quantized training in NVFP4 via a novel unbiased quantization routine for micro-scaled formats, called MS-EDEN, that has more than 2x lower quantization error than SR. We integrate it into a novel fully-NVFP4 quantization scheme for linear layers, called Quartet II. We show analytically that Quartet II achieves consistently better gradient estimation across all major matrix multiplications, both on the forward and on the backward passes. In addition, our proposal synergizes well with recent training improvements aimed specifically at NVFP4. We further validate Quartet II on end-to-end LLM training with up to 1.9B parameters on 38B tokens. We provide kernels for execution on NVIDIA Blackwell GPUs with up to 4.2x speedup over BF16. Our code is available at https://github.com/IST-DASLab/Quartet-II .

翻译：由NVIDIA Blackwell GPU硬件支持的NVFP4低精度格式，首次有望实现大规模模型（如大语言模型）端到端的全量化预训练。然而，现有的量化训练方法仍会牺牲该格式的部分表示能力，以换取通过随机舍入（SR）进行更精确的无偏量化梯度估计，导致相对于标准FP16和FP8训练存在明显的精度损失。本文通过一种名为MS-EDEN的新型微尺度格式无偏量化例程，改进了NVFP4量化训练的技术水平，其量化误差比SR低2倍以上。我们将其集成到一种名为Quartet II的新型全NVFP4线性层量化方案中。通过分析，我们证明Quartet II在前向传播和反向传播的所有主要矩阵乘法中均能实现更优的梯度估计。此外，我们的方案与近期专门针对NVFP4的训练改进技术能良好协同。我们进一步在高达19亿参数、380亿token的端到端大语言模型训练上验证了Quartet II。我们提供了可在NVIDIA Blackwell GPU上执行的核函数，相比BF16实现了高达4.2倍的加速。代码发布于https://github.com/IST-DASLab/Quartet-II。