The growing demand for Large Language Models (LLMs) in applications such as content generation, intelligent chatbots, and sentiment analysis poses considerable challenges for LLM service providers. To efficiently use GPU resources and boost throughput, batching multiple requests has emerged as a popular paradigm; to further speed up batching, LLM quantization techniques reduce memory consumption and increase computing capacity. However, prevalent quantization schemes (e.g., 8-bit weight-activation quantization) cannot fully leverage the capabilities of modern GPUs, such as 4-bit integer operators, resulting in sub-optimal performance. To maximize LLMs' serving throughput, we introduce Atom, a low-bit quantization method that achieves high throughput improvements with negligible accuracy loss. Atom significantly boosts serving throughput by using low-bit operators and considerably reduces memory consumption via low-bit quantization. It attains high accuracy by applying a novel mixed-precision and fine-grained quantization process. We evaluate Atom on 4-bit weight-activation quantization setups in the serving context. Atom improves end-to-end throughput by up to $7.73\times$ compared to the FP16 and by $2.53\times$ compared to INT8 quantization, while maintaining the same latency target.

翻译：大语言模型在内容生成、智能对话系统及情感分析等应用中的需求日益增长，给LLM服务提供商带来显著挑战。为高效利用GPU资源并提升吞吐量，批处理请求已成为主流范式；为进一步加速批处理过程，LLM量化技术通过降低内存消耗、增强计算能力实现优化。然而，现有量化方案（如8位权重-激活量化）无法充分挖掘现代GPU的计算能力（如4位整数算子），导致性能欠佳。为最大化LLM服务吞吐量，我们提出Atom——一种低比特量化方法，能在保持精度损失可忽略的前提下实现高吞吐量提升。Atom通过采用低比特算子显著提升服务吞吐量，并通过低比特量化大幅减少内存消耗。该方法通过创新的混合精度与细粒度量化流程实现高精度。我们在服务场景下的4位权重-激活量化配置中评估了Atom。与FP16相比，Atom将端到端吞吐量提升高达7.73倍，相较于INT8量化提升2.53倍，同时维持相同延迟目标。