Several post-training quantization methods have been applied to large language models (LLMs), and have been shown to perform well down to 8-bits. We find that these methods break down at lower bit precision, and investigate quantization aware training for LLMs (LLM-QAT) to push quantization levels even further. We propose a data-free distillation method that leverages generations produced by the pre-trained model, which better preserves the original output distribution and allows quantizing any generative model independent of its training data, similar to post-training quantization methods. In addition to quantizing weights and activations, we also quantize the KV cache, which is critical for increasing throughput and support long sequence dependencies at current model sizes. We experiment with LLaMA models of sizes 7B, 13B, and 30B, at quantization levels down to 4-bits. We observe large improvements over training-free methods, especially in the low-bit settings.

翻译：多种后训练量化方法已被应用于大语言模型（LLMs），并展现出在8比特精度下的良好性能。然而我们发现这些方法在更低比特精度下性能显著下降，因此我们探索面向大语言模型的量化感知训练（LLM-QAT）以进一步突破量化极限。我们提出一种无数据蒸馏方法，利用预训练模型生成的样本来更好地保留原始输出分布，该方法可像后训练量化方法一样，在无需依赖训练数据的情况下量化任意生成式模型。除量化权重和激活值外，我们还对关键性KV缓存进行量化——这对于在当前模型规模下提升吞吐量并支持长序列依赖至关重要。我们在LLaMA系列模型（7B、13B和30B参数规模）上进行了低至4比特的量化实验。结果表明，相较于无训练方法，本方法在低比特场景下取得了显著改进。