Large language models (LLMs) show excellent performance but are compute- and memory-intensive. Quantization can reduce memory and accelerate inference. However, existing methods cannot maintain accuracy and hardware efficiency at the same time. We propose SmoothQuant, a training-free, accuracy-preserving, and general-purpose post-training quantization (PTQ) solution to enable 8-bit weight, 8-bit activation (W8A8) quantization for LLMs. Based on the fact that weights are easy to quantize while activations are not, SmoothQuant smooths the activation outliers by offline migrating the quantization difficulty from activations to weights with a mathematically equivalent transformation. SmoothQuant enables an INT8 quantization of both weights and activations for all the matrix multiplications in LLMs, including OPT, BLOOM, GLM, MT-NLG, and LLaMA family. We demonstrate up to 1.56x speedup and 2x memory reduction for LLMs with negligible loss in accuracy. SmoothQuant enables serving 530B LLM within a single node. Our work offers a turn-key solution that reduces hardware costs and democratizes LLMs. Code is available at https://github.com/mit-han-lab/smoothquant.

翻译：大型语言模型（LLMs）性能卓越但计算与内存消耗巨大。量化技术可降低内存占用并加速推理，然而现有方法难以同时兼顾精度与硬件效率。本文提出SmoothQuant——一种无需重新训练、保持精度且通用的训练后量化（PTQ）方案，实现LLMs的8位权重与8位激活值（W8A8）量化。基于权重易量化而激活值难量化的特性，SmoothQuant通过数学等价变换，将量化难度从激活值离线迁移至权重，从而平滑激活值中的异常点。该方案支持对所有矩阵乘法（涵盖OPT、BLOOM、GLM、MT-NLG及LLaMA系列模型）进行INT8权重与激活值量化。实验表明，在精度损失可忽略的前提下，LLMs推理速度提升达1.56倍，内存占用降低2倍。SmoothQuant使单个节点即可服务530B参数的大语言模型，提供降低硬件成本、推动LLMs普及的一站式解决方案。代码开源地址：https://github.com/mit-han-lab/smoothquant。