In the complex domain of large language models (LLMs), striking a balance between computational efficiency and maintaining model quality is a formidable challenge. Navigating the inherent limitations of uniform quantization, particularly when dealing with outliers, and motivated by the launch of NVIDIA's H100 hardware, this study delves into the viability of floating-point (FP) quantization, particularly focusing on FP8 and FP4, as a potential solution. Our comprehensive investigation reveals that for LLMs, FP8 activation consistently outshines its integer (INT8) equivalent, with the performance edge becoming more noticeable in models possessing parameters beyond one billion. For weight quantization, our findings indicate that FP4 exhibits comparable, if not superior, performance to INT4, simplifying deployment on FP-supported hardware like H100. To mitigate the overhead from precision alignment caused by the disparity between weights and activations, we propose two scaling constraints for weight quantization that negligibly impact the performance compared to the standard W4A8 model. We additionally enhance our quantization methods by integrating the Low Rank Compensation (LoRC) strategy, yielding improvements especially in smaller models. The results of our investigation emphasize the immense potential of FP quantization for LLMs, paving the way for high-efficiency deployment in resource-limited settings.

翻译：在大语言模型（LLMs）的复杂领域中，在计算效率与模型质量之间取得平衡是一项艰巨挑战。针对均匀量化在处理异常值时固有的局限性，并受NVIDIA H100硬件推出的启发，本研究深入探究了浮点（FP）量化的可行性，尤其聚焦FP8与FP4格式作为潜在解决方案。我们的全面研究表明：对于LLMs，FP8激活值在性能上始终优于其整数量化（INT8）对应方案，且当模型参数规模超过十亿时，该性能优势更为显著。在权重量化方面，实验结果表明FP4不仅性能与INT4相当，甚至更优，从而简化了在H100等支持浮点运算的硬件上的部署过程。为缓解因权重与激活值精度差异导致的校准开销问题，我们针对权重量化提出了两种缩放约束条件，相较于标准W4A8模型，其对性能的影响微乎其微。此外，我们通过集成低秩补偿（LoRC）策略进一步优化了量化方法，尤其在较小模型中取得了显著改进。本研究的成果凸显了浮点量化在LLMs领域的巨大潜力，为资源受限场景下的高效部署铺平了道路。