Large language model (LLM) inference demands significant amount of computation and memory, especially in the key attention mechanism. While techniques, such as quantization and acceleration algorithms, like FlashAttention, have improved efficiency of the overall inference, they address different aspects of the problem: quantization focuses on weight-activation operations, while FlashAttention improves execution but requires high-precision formats. Recent Key-value (KV) cache quantization reduces memory bandwidth but still needs floating-point dequantization for attention operation. We present TurboAttention, a comprehensive approach to enable quantized execution of attention that simultaneously addresses both memory and computational efficiency. Our solution introduces two key innovations: FlashQ, a headwise attention quantization technique that enables both compression of KV cache and quantized execution of activation-activation multiplication, and Sparsity-based Softmax Approximation (SAS), which eliminates the need for dequantization to FP32 during exponentiation operation in attention. Experimental results demonstrate that TurboAttention achieves 1.2-1.8x speedup in attention, reduces the KV cache size by over 4.4x, and enables up to 2.37x maximum throughput over the FP16 baseline while outperforming state-of-the-art quantization and compression techniques across various datasets and models.

翻译：大语言模型（LLM）推理需要大量的计算与内存资源，其核心注意力机制尤为关键。尽管量化技术和FlashAttention等加速算法提升了整体推理效率，但它们分别针对不同层面：量化主要处理权重-激活运算，而FlashAttention虽改善了执行效率却依赖高精度格式。近期的键值（KV）缓存量化技术虽降低了内存带宽需求，但在注意力运算中仍需进行浮点反量化。本文提出TurboAttention——一种实现注意力量化执行的综合性方案，可同步提升内存与计算效率。该方案包含两项核心创新：FlashQ（一种支持KV缓存压缩及激活-激活乘法量化执行的头部注意力量化技术）与基于稀疏性的Softmax近似方法（SAS），后者可在注意力指数运算中避免向FP32格式的反量化操作。实验结果表明，TurboAttention在注意力计算上实现1.2-1.8倍加速，将KV缓存大小缩减4.4倍以上，并在FP16基准上达到最高2.37倍的吞吐量提升，同时在多种数据集与模型上优于当前最先进的量化与压缩技术。