The Transformer architecture has significantly advanced natural language processing (NLP) and has been foundational in developing large language models (LLMs) such as LLaMA and OPT, which have come to dominate a broad range of NLP tasks. Despite their superior accuracy, LLMs present unique challenges in practical inference, concerning the compute and memory-intensive nature. Thanks to the autoregressive characteristic of LLM inference, KV caching for the attention layers in Transformers can effectively accelerate LLM inference by substituting quadratic-complexity computation with linear-complexity memory accesses. Yet, this approach requires increasing memory as demand grows for processing longer sequences. The overhead leads to reduced throughput due to I/O bottlenecks and even out-of-memory errors, particularly on resource-constrained systems like a single commodity GPU. In this paper, we propose ALISA, a novel algorithm-system co-design solution to address the challenges imposed by KV caching. On the algorithm level, ALISA prioritizes tokens that are most important in generating a new token via a Sparse Window Attention (SWA) algorithm. SWA introduces high sparsity in attention layers and reduces the memory footprint of KV caching at negligible accuracy loss. On the system level, ALISA employs three-phase token-level dynamical scheduling and optimizes the trade-off between caching and recomputation, thus maximizing the overall performance in resource-constrained systems. In a single GPU-CPU system, we demonstrate that under varying workloads, ALISA improves the throughput of baseline systems such as FlexGen and vLLM by up to 3X and 1.9X, respectively.

翻译：Transformer架构显著推动了自然语言处理（NLP）的发展，并成为LLaMA、OPT等大型语言模型（LLM）的基础，这些模型在广泛的NLP任务中占据主导地位。尽管LLM具备卓越的准确性，但其实际推理过程因计算和内存密集型特性而面临独特挑战。得益于LLM推理的自回归特性，Transformer注意力层中的KV缓存可以通过将二次复杂度计算转化为线性复杂度内存访问来有效加速推理。然而，随着处理更长序列的需求增长，该方法需要不断增加内存。这种开销会导致I/O瓶颈造成的吞吐量下降，甚至在资源受限系统（如单个商用GPU）中引发内存溢出错误。本文提出ALISA——一种新颖的算法-系统协同设计方案，以应对KV缓存带来的挑战。在算法层面，ALISA通过稀疏窗口注意力（SWA）算法优先处理对生成新令牌最重要的令牌。SWA在注意力层中引入高稀疏性，以可忽略的精度损失减少KV缓存的内存占用。在系统层面，ALISA采用三阶段令牌级动态调度，并优化缓存与重计算之间的权衡，从而在资源受限系统中最大化整体性能。在单GPU-CPU系统中，我们证明在不同工作负载下，ALISA相比FlexGen和vLLM等基线系统的吞吐量分别提升高达3倍和1.9倍。