FP8 formats are gaining popularity to boost the computational efficiency for training and inference of large deep learning models. Their main challenge is that a careful choice of scaling is needed to prevent degradation due to the reduced dynamic range compared to higher-precision formats. Although there exists ample literature about selecting such scalings for INT formats, this critical aspect has yet to be addressed for FP8. This paper presents a methodology to select the scalings for FP8 linear layers, based on dynamically updating per-tensor scales for the weights, gradients and activations. We apply this methodology to train and validate large language models of the type of GPT and Llama 2 using FP8, for model sizes ranging from 111M to 70B. To facilitate the understanding of the FP8 dynamics, our results are accompanied by plots of the per-tensor scale distribution for weights, activations and gradients during both training and inference.

翻译：FP8格式正日益被广泛采用，以提升大型深度学习模型训练与推理的计算效率。其主要挑战在于，相较于高精度格式，其动态范围缩减，需谨慎选择缩放因子以防止性能退化。尽管已有大量文献探讨整数（INT）格式的缩放选择，但FP8格式中这一关键问题尚未得到充分研究。本文提出了一种基于动态更新权重、梯度及激活值逐张量缩放因子的FP8线性层缩放选择方法。我们采用该方法，对111M至70B参数规模的GPT与Llama 2类大语言模型进行FP8训练与验证。为便于理解FP8的动态特性，实验结果附带了训练与推理过程中权重、激活值及梯度逐张量缩放因子的分布曲线图。