As the demand for deep learning grows, cost reduction through quantization has become essential for both training and inference. In 2022, the Open Compute Project (OCP) consortium standardized narrow precision formats for deep learning, called the microscaling (MX) format. The MX format is a hardware-friendly dynamic quantization scheme that effectively reduces the data size by sharing an 8-bit exponent across multiple operands. The MX format can be categorized into two types with their own strengths: (i) MXINT which focuses on a high precision consisting only of mantissa bits and (ii) MXFP which focuses on a wider dynamic range by allowing local exponent bits. In this work, we present a versatile MXFP format, called MX-SAFE (MXSF in short), that adaptively uses two modes, i.e., a wider mantissa mode (FP8 E2M5) and a subnormal FP mode (FP5 E3M2), to support both training and direct-cast inference. Furthermore, we propose a tile-based block design to increase hardware efficiency by reducing the burden of re-quantization process during the training with the MXSF format. Owing to the use of the proposed MXSF format, 0.05%/11.1% and 3.55%/3.57% improvements in accuracy, on average, for inference/full-training compared to MXFP8 E2M5 and MXFP8 E4M3 are observed, respectively. Moreover, we present a training-inference accelerator that supports the MXSF format and it achieves similar accuracy to the BF16 baseline while using 24.9% less total energy consumption.

翻译：随着深度学习需求的增长，通过量化降低计算成本已成为训练和推理过程中不可或缺的技术。2022年，开放计算项目（OCP）联盟为深度学习标准化了窄精度格式，即微缩放（MX）格式。该格式是一种硬件友好的动态量化方案，通过跨多个操作数共享8位指数有效减少数据规模。MX格式可分为两类，各具优势：（i）MXINT仅包含尾数位以实现高精度，（ii）MXFP通过允许本地指数位实现更广泛的动态范围。本文提出一种多功能MXFP格式——MX-SAFE（简称MXSF），该格式自适应采用两种模式：宽尾数模式（FP8 E2M5）和次正规FP模式（FP5 E3M2），同时支持训练与直接转换推理。此外，我们提出基于分块的块设计，通过减少MXSF格式训练过程中重量化过程的负担提升硬件效率。采用所提出的MXSF格式后，在推理/全训练任务中，相比MXFP8 E2M5和MXFP8 E4M3，平均准确率分别提升0.05%/11.1%和3.55%/3.57%。同时，我们提出支持MXSF格式的训练-推理加速器，其在总能耗降低24.9%的条件下实现了与BF16基线相当的准确率。