Transformer-based models, such as BERT, have been widely applied in a wide range of natural language processing tasks. However, one inevitable side effect is that they require massive memory storage and inference cost when deployed in production. Quantization is one of the popularized ways to alleviate the cost. However, the previous 8-bit quantization strategy based on INT8 data format either suffers from the degradation of accuracy in a Post-Training Quantization (PTQ) fashion or requires an expensive Quantization-Aware Training (QAT) process. Recently, a new numeric format FP8 (i.e. floating-point of 8-bits) has been proposed and supported in commercial AI computing platforms such as H100. In this paper, we empirically validate the effectiveness of FP8 as a way to do Post-Training Quantization without significant loss of accuracy, with a simple calibration and format conversion process. We adopt the FP8 standard proposed by NVIDIA Corp. (2022) in our extensive experiments of BERT variants on GLUE and SQuAD v1.1 datasets, and show that PTQ with FP8 can significantly improve the accuracy upon that with INT8, to the extent of the full-precision model.

翻译：基于Transformer的模型（如BERT）已广泛应用于各类自然语言处理任务。然而，一个不可避免的副作用是，当这些模型部署到生产环境中时，需要大量的内存存储和推理成本。量化是降低成本的常见方法之一。然而，先前基于INT8数据格式的8位量化策略，要么在训练后量化（PTQ）方式下出现精度损失，要么需要成本高昂的量化感知训练（QAT）过程。最近，一种新型数值格式FP8（即8位浮点数）被提出并得到H100等商业AI计算平台的支持。本文通过简单的校准和格式转换过程，实证验证了FP8作为一种训练后量化方法的有效性，且不会造成显著的精度损失。我们采用NVIDIA公司（2022年）提出的FP8标准，在GLUE和SQuAD v1.1数据集上对BERT变体进行了大量实验，结果表明，相较于INT8，采用FP8的PTQ可将精度显著提升至全精度模型水平。