Large language models (LLMs) have demonstrated impressive abilities in various domains while the inference cost is expensive. The state-of-the-art methods use 2-bit quantization for mainstream LLMs. However, challenges still exist: (1) Nonnegligible accuracy loss for 2-bit quantization. Weights are quantized by groups, while the ranges of weights are large in some groups, resulting in large quantization errors and nonnegligible accuracy loss (e.g. >3% for Llama2-7b with 2-bit quantization in GPTQ and Greenbit). (2) Limited accuracy improvement by adding 4-bit weights. Increasing 10% extra average bit more 4-bit weights only leads to <0.5% accuracy improvement on a quantized Llama2-7b. (3) Time-consuming dequantization operations on GPUs. The dequantization operations lead to >50% execution time, hindering the potential of reducing LLM inference cost. To tackle these challenges, we propose the following techniques: (1) We only quantize a small fraction of groups with the larger range using 4-bit with memory alignment consideration on GPUs. (2) We point out that the distribution of the sparse outliers with larger weights is different in 2-bit and 4-bit groups, and only a small fraction of outliers require 16-bit quantization. Such design leads to >0.5% accuracy improvement with <3% average increased bit for Llama2-7b. (3) We design the asynchronous dequantization on GPUs, leading to up to 3.92X speedup. We conduct extensive experiments on different model families and model sizes. We achieve 2.85-bit for each weight and the end-to-end speedup for Llama2-7b is 1.74X over the original model, and we reduce both runtime cost and hardware cost by up to 2.70X and 2.81X with less GPU requirements.

翻译：大语言模型（LLMs）在各个领域展现出令人印象深刻的能力，但其推理成本高昂。当前主流方法采用2比特量化处理大语言模型，然而仍面临以下挑战：（1）2比特量化存在不可忽略的精度损失。权重按组量化，但部分组的权重范围过大，导致量化误差显著，造成不可忽略的精度损失（例如，GPTQ和Greenbit对Llama2-7b进行2比特量化时，精度损失超过3%）。（2）增加4比特权重的精度提升有限。在量化后的Llama2-7b模型中，额外增加10%的平均比特数以使用4比特权重，仅带来不足0.5%的精度提升。（3）GPU上反量化操作耗时严重。反量化操作占执行时间的50%以上，限制了降低LLM推理成本的潜力。为应对这些挑战，我们提出以下技术：（1）仅对少数权重范围较大的组进行4比特量化，并考虑GPU内存对齐。（2）我们发现2比特与4比特组中权重较大的稀疏异常值分布存在差异，仅需对少量异常值进行16比特量化。该设计使Llama2-7b在平均比特增加不足3%的情况下，实现超过0.5%的精度提升。（3）我们设计了GPU上的异步反量化方法，最高可实现3.92倍加速。我们在不同模型家族和模型规模上进行了大量实验，最终实现每个权重仅需2.85比特，Llama2-7b的端到端加速比达到原始模型的1.74倍，同时运行时成本和硬件成本分别降低2.70倍和2.81倍，且所需GPU资源更少。