The unprecedented growth in DNN model complexity, size, and amount of training data has led to a commensurate increase in demand for computing and a search for minimal encoding. Recent research advocates Hybrid Block Floating Point (HBFP) to minimize silicon provisioning in accelerators by converting the majority of arithmetic operations in training to 8-bit fixed point. In this paper, we perform a full-scale exploration of the HBFP design space using mathematical tools to study the interplay among various parameters and identify opportunities for even smaller encodings across layers and epochs. Based on our findings, we propose Accuracy Boosters, an epoch-driven mixed-mantissa HBFP technique that uses 6-bit mantissas only in the last epoch and first/last layers, and 4-bit mantissas for $99.7\%$ of all other arithmetic operations in training. Using analytic models, we show Accuracy Boosters enable increasing arithmetic density for an HBFP training accelerator by up to $21.3\times$ compared to FP32 and up to $4.4\times$ compared to another SOTA format Bfloat16, while preserving or outperforming FP32 accuracy.

翻译：深度学习模型在复杂性、规模和训练数据量上的空前增长，导致计算需求相应激增，并促使研究者寻求最小化编码方案。近期研究提出混合块浮点技术，通过将训练中大部分算术运算转换为8位定点数，以最小化加速器的硅基硬件配置。本文利用数学工具对混合块浮点设计空间进行全尺度探索，系统研究各参数间的相互作用，并识别跨层与历元间实现更小编码的潜在机遇。基于研究发现，我们提出精度提升器——一种历元驱动混合尾数块浮点技术，该技术仅在最后一个历元及首尾层使用6位尾数，而对训练中其余99.7%的算术运算采用4位尾数。通过解析模型分析，我们证明精度提升器可使混合块浮点训练加速器的算术密度相较FP32提升至高21.3倍，相较另一当前最优格式Bfloat16提升至高4.4倍，同时保持或超越FP32的精度水平。