Six-bit quantization (FP6) can effectively reduce the size of large language models (LLMs) and preserve the model quality consistently across varied applications. However, existing systems do not provide Tensor Core support for FP6 quantization and struggle to achieve practical performance improvements during LLM inference. It is challenging to support FP6 quantization on GPUs due to (1) unfriendly memory access of model weights with irregular bit-width and (2) high runtime overhead of weight de-quantization. To address these problems, we propose TC-FPx, the first full-stack GPU kernel design scheme with unified Tensor Core support of float-point weights for various quantization bit-width. We integrate TC-FPx kernel into an existing inference system, providing new end-to-end support (called FP6-LLM) for quantized LLM inference, where better trade-offs between inference cost and model quality are achieved. Experiments show that FP6-LLM enables the inference of LLaMA-70b using only a single GPU, achieving 1.69x-2.65x higher normalized inference throughput than the FP16 baseline. The source code will be publicly available soon.

翻译：六位量化（FP6）能够有效减小大语言模型的体积，并在各种应用中持续保持模型质量。然而，现有系统未提供针对FP6量化的张量核心支持，且难以在大语言模型推理过程中实现实际性能提升。在GPU上支持FP6量化面临两大挑战：（1）不规则位宽模型权重导致的存储访问不友好；（2）权重反量化带来的高运行时开销。为解决这些问题，我们提出TC-FPx——首个统一支持浮点权重的全栈GPU内核设计方案，可兼容多种量化位宽。我们将TC-FPx内核集成至现有推理系统，为量化大语言模型推理提供全新端到端支持（称为FP6-LLM），在推理成本与模型质量之间实现更优平衡。实验表明，FP6-LLM仅需单张GPU即可运行LLaMA-70b模型，其归一化推理吞吐量相比FP16基线提升1.69倍至2.65倍。源代码即将公开发布。