Large language models (LLMs) have shown remarkable capabilities in various tasks. However their huge model size and the consequent demand for computational and memory resources also pose challenges to model deployment. Currently, 4-bit post-training quantization (PTQ) has achieved some success in LLMs, reducing the memory footprint by approximately 75% compared to FP16 models, albeit with some accuracy loss. In this paper, we propose SmoothQuant+, an accurate and efficient 4-bit weight-only PTQ that requires no additional training, which enables lossless in accuracy for LLMs for the first time. Based on the fact that the loss of weight quantization is amplified by the activation outliers, SmoothQuant+ smoothes the activation outliers by channel before quantization, while adjusting the corresponding weights for mathematical equivalence, and then performs group-wise 4-bit weight quantization for linear layers. We have integrated SmoothQuant+ into the vLLM framework, an advanced high-throughput inference engine specially developed for LLMs, and equipped it with an efficient W4A16 CUDA kernels, so that vLLM can seamlessly support SmoothQuant+ 4-bit weight quantization. Our results show that, with SmoothQuant+, the Code Llama-34B model can be quantized and deployed on a A100 40GB GPU, achieving lossless accuracy and a throughput increase of 1.9 to 4.0 times compared to the FP16 model deployed on two A100 40GB GPUs. Moreover, the latency per token is only 68% of the FP16 model deployed on two A100 40GB GPUs. This is the state-of-the-art 4-bit weight quantization for LLMs as we know.

翻译：大语言模型（LLMs）在各类任务中展现出卓越能力，但其庞大的模型尺寸及随之而来的计算与内存资源需求给模型部署带来挑战。目前，4比特训练后量化（PTQ）已在大语言模型中取得一定成功，相较FP16模型可减少约75%内存占用，但会伴随精度损失。本文提出SmoothQuant+——一种无需额外训练的精确高效纯权重量化方案，首次实现大语言模型在4比特PTQ下的无损精度。基于权重量化损失会被激活值异常放大这一事实，SmoothQuant+在量化前通过通道维度平滑激活值异常，同时数学等价调整对应权重，再对线性层执行分组4比特权重量化。我们将SmoothQuant+集成至专为大语言模型开发的高吞吐量推理引擎vLLM框架，并为其定制高效W4A16 CUDA内核，使vLLM能无缝支持SmoothQuant+的4比特权重量化。实验表明，采用SmoothQuant+可将Code Llama-34B模型量化部署于单块A100 40GB GPU，实现无损精度，吞吐量较双A100 40GB GPU部署的FP16模型提升1.9至4.0倍；每个token的延迟仅为双A100 40GB GPU部署FP16模型的68%。据我们所知，这是当前大语言模型领域最先进的4比特权重量化方案。