Large language models (LLMs) show excellent performance but are compute- and memory-intensive. Quantization can reduce memory and accelerate inference. However, for LLMs beyond 100 billion parameters, existing methods cannot maintain accuracy or do not run efficiently on hardware. We propose SmoothQuant, a training-free, accuracy-preserving, and general-purpose post-training quantization (PTQ) solution to enable 8-bit weight, 8-bit activation (W8A8) quantization for LLMs. Based on the fact that weights are easy to quantize while activations are not, SmoothQuant smooths the activation outliers by offline migrating the quantization difficulty from activations to weights with a mathematically equivalent transformation. SmoothQuant enables an INT8 quantization of both weights and activations for all the matrix multiplications in LLMs, including OPT-175B, BLOOM-176B, GLM-130B, and MT-NLG 530B. SmoothQuant has better hardware efficiency than existing techniques. We demonstrate up to 1.56x speedup and 2x memory reduction for LLMs with negligible loss in accuracy. We integrate SmoothQuant into FasterTransformer, a state-of-the-art LLM serving framework, and achieve faster inference speed with half the number of GPUs compared to FP16, enabling the serving of a 530B LLM within a single node. Our work offers a turn-key solution that reduces hardware costs and democratizes LLMs. Code is available at https://github.com/mit-han-lab/smoothquant.

翻译：大语言模型（LLMs）虽展现卓越性能，但计算与内存开销巨大。量化技术可降低内存占用并加速推理。然而，针对参数超千亿的LLMs，现有方法无法维持精度或在硬件上高效运行。我们提出SmoothQuant——一种无需训练、保持精度且通用的训练后量化（PTQ）方案，可实现LLMs的8位权重与8位激活（W8A8）量化。基于权重易量化而激活难量化的特性，SmoothQuant通过数学等价的变换，将量化难度从激活离线迁移至权重，从而平滑激活值离群点。该方案可对LLMs（包括OPT-175B、BLOOM-176B、GLM-130B和MT-NLG 530B）中所有矩阵乘法实现权重与激活的INT8量化。相比现有技术，SmoothQuant具有更优的硬件效率。我们证明该方案能在精度损失可忽略的前提下，为LLMs带来最高1.56倍加速与2倍内存压缩。通过将SmoothQuant集成至先进LLM服务框架FasterTransformer，相比FP16方案可减少一半GPU数量并实现更快推理速度，使530B参数LLM可在单节点内完成服务。本工作提供了一种即插即用方案，有效降低硬件成本并推动LLMs民主化。代码开源地址：https://github.com/mit-han-lab/smoothquant。