Increasing the number of parameters in large language models (LLMs) usually improves performance in downstream tasks but raises compute and memory costs, making deployment difficult in resource-limited settings. Quantization techniques, which reduce the bits needed for model weights or activations with minimal performance loss, have become popular due to the rise of LLMs. However, most quantization studies use pre-trained LLMs, and the impact of quantization on instruction-tuned LLMs and the relationship between perplexity and benchmark performance of quantized LLMs are not well understood. Evaluation of quantized LLMs is often limited to language modeling and a few classification tasks, leaving their performance on other benchmarks unclear. To address these gaps, we propose a structured evaluation framework consisting of three critical dimensions: (1) knowledge \& capacity, (2) alignment, and (3) efficiency, and conduct extensive experiments across ten diverse benchmarks. Our experimental results indicate that LLMs with 4-bit quantization can retain performance comparable to their non-quantized counterparts, and perplexity can serve as a proxy metric for quantized LLMs on most benchmarks. Furthermore, quantized LLMs with larger parameter scales can outperform smaller LLMs. Despite the memory savings achieved through quantization, it can also slow down the inference speed of LLMs. Consequently, substantial engineering efforts and hardware support are imperative to achieve a balanced optimization of decoding speed and memory consumption in the context of quantized LLMs.

翻译：增加大型语言模型（LLM）的参数数量通常能提升下游任务性能，但会带来计算和内存成本的增加，使其在资源受限环境中的部署变得困难。量化技术通过减少模型权重或激活值所需的比特数，在性能损失最小化的前提下降低资源消耗，随着LLM的兴起而日益流行。然而，大多数量化研究采用预训练LLM进行实验，量化对指令微调LLM的影响以及困惑度与量化LLM基准性能之间的关系尚未得到充分理解。量化LLM的评估通常局限于语言建模和少量分类任务，导致其在不同基准上的表现尚不明确。为填补这些空白，我们提出一个结构化评估框架，涵盖三个关键维度：（1）知识能力、（2）对齐性和（3）效率，并在十个不同基准上开展广泛实验。实验结果表明，采用4比特量化的LLM可保持与非量化版本相当的性能，且困惑度可作为量化LLM在多数基准上的代理指标。此外，参数量更大的量化LLM可超越较小规模的LLM。尽管量化能节省内存，但也会降低LLM的推理速度。因此，需要大量的工程努力和硬件支持，才能在量化LLM的解码速度与内存消耗之间实现平衡优化。