Generative Large Language Models (LLMs) have demonstrated remarkable results for a wide range of tasks. However, deploying these models for inference has been a significant challenge due to their unprecedented resource requirements. This has forced existing deployment frameworks to use multi-GPU inference pipelines, which are often complex and costly, or to use smaller and less performant models. In this work, we demonstrate that the main bottleneck for generative inference with LLMs is memory bandwidth, rather than compute, specifically for single batch inference. While quantization has emerged as a promising solution by representing model weights with reduced precision, previous efforts have often resulted in notable performance degradation. To address this, we introduce SqueezeLLM, a post-training quantization framework that not only enables lossless compression to ultra-low precisions of up to 3-bit, but also achieves higher quantization performance under the same memory constraint. Our framework incorporates two novel ideas: (i) sensitivity-based non-uniform quantization, which searches for the optimal bit precision assignment based on second-order information; and (ii) the Dense-and-Sparse decomposition that stores outliers and sensitive weight values in an efficient sparse format. When applied to the LLaMA models, our 3-bit quantization significantly reduces the perplexity gap from the FP16 baseline by up to 2.1x as compared to the state-of-the-art methods with the same memory requirement. Furthermore, when deployed on an A6000 GPU, our quantized models achieve up to 2.3x speedup compared to the baseline. Our code is open-sourced and available online.

翻译：生成式大语言模型（LLMs）已在多种任务中展现出显著效果。然而，由于这些模型前所未有的资源需求，将其部署用于推理一直是一项重大挑战。这迫使现有部署框架要么使用复杂且昂贵的多GPU推理流水线，要么采用性能较差的小型模型。在本工作中，我们证明LLMs生成式推理的主要瓶颈在于内存带宽而非计算能力，尤其是在单批次推理场景中。尽管通过降低模型权重精度表示的量化方法已成为一种有前景的解决方案，但先前的工作往往导致显著的性能下降。为解决此问题，我们提出SqueezeLLM后训练量化框架，该框架不仅能实现低至3位精度的无损压缩，还能在相同内存约束下获得更高的量化性能。我们的框架包含两个创新思想：（i）基于敏感度的非均匀量化——基于二阶信息搜索最优位宽分配；（ii）稠密-稀疏分解——以高效稀疏格式存储离群值和敏感权重。将本方法应用于LLaMA模型时，在相同内存需求下，我们的3位量化相较于最先进方法将FP16基线的困惑度差距最多缩小2.1倍。此外，在A6000 GPU上部署时，我们的量化模型相比基线实现最高2.3倍加速。本方法的代码已开源并可供在线获取。