Model quantization represents both parameters (weights) and intermediate values (activations) in a more compact format, thereby directly reducing both computational and memory cost in hardware. The quantization of recent large language models (LLMs) faces challenges to achieve competitive memory density compared to other models such as convolutional neural networks, since values in LLMs require larger dynamic ranges. Current hardware can expedite computation for LLMs using compact numerical formats such as low-bitwidth integers or floating-point numbers. Each has advantages: integer operations simplify circuit design, whereas floating-point calculations can enhance accuracy when a wider dynamic range is required. In this work, we seek an efficient data format that combines the best of both worlds: Microscaling (MX) formats. MX formats are efficient data formats that achieve both large dynamic ranges and high memory density. In this paper, we propose a compiler named MASE for exploring mixed-precision MX formats on dataflow hardware accelerators for LLM inference. Our main contributions are twofold. First, we propose a novel orchestration abstraction to explore both software and hardware optimizations with new data formats. Second, MASE achieves LLM inference at an average precision of 4-bits, with minimal to no accuracy degradation. To our knowledge, MASE represents the first effort to harness fine-grain multi-precision MX formats in the design of LLM hardware accelerators. Over a range of LLMs and datasets, MASE achieves an average improvement of 24% in $\Delta$ accuracy with an overhead of only 3% in energy efficiency compared to designs using 8-bit fixed-point numbers.

翻译：模型量化将参数（权重）和中间值（激活值）以更紧凑的格式表示，从而直接降低硬件中的计算和内存成本。与卷积神经网络等其他模型相比，当前大语言模型的量化在实现有竞争力的内存密度方面面临挑战，因为大语言模型中的值需要更大的动态范围。现有硬件可使用低比特整数或浮点数等紧凑数值格式加速大语言模型计算。两者各有优势：整数运算简化电路设计，而浮点计算在需要更宽动态范围时可提升精度。在本研究中，我们寻求一种结合两者优势的高效数据格式：微缩放格式。微缩放格式是一种高效的数据格式，既能实现大动态范围，又能实现高内存密度。本文提出名为MASE的编译器，用于探索数据流硬件加速器上大语言模型推理的混合精度微缩放格式。我们的主要贡献有两方面：首先，提出一种新颖的编排抽象，以探索结合新数据格式的软件和硬件优化；其次，MASE在平均4比特精度下实现大语言模型推理，且精度退化极小甚至无退化。据我们所知，MASE是首个在大语言模型硬件加速器设计中利用细粒度多精度微缩放格式的尝试。在多种大语言模型和数据集上，与使用8比特定点数的设计相比，MASE在能量效率仅增加3%的开销下，平均实现了24%的$\Delta$精度提升。