Long-context LLM serving is bottlenecked by the cost of attending over ever-growing KV caches. Dynamic sparse attention promises relief by accessing only a small, query-dependent subset of the KV state per decoding step and extending the KV storage to CPU memory. In practice, however, these algorithmic savings rarely translate into end-to-end system-level gains because sparse methods typically operate at different granularities and thus rely on ad hoc, per-algorithm implementations. At the same time, hierarchical KV storage introduces a new systems bottleneck: retrieving fine-grained, irregular KV subsets across the GPU-CPU boundary can easily erase the benefits of sparsity. We present SPIN, a sparse-attention-aware inference framework that co-designs the execution pipeline with hierarchical KV storage through three techniques: (1) a unified partition abstraction that maps different sparsity granularities onto a shared page-based KV substrate; (2) a locality-aware KV cache manager that dynamically sizes per-request HBM budgets and uses a GPU-friendly bucketed LRU policy to cut PCIe round-trips; and (3) a two-level hierarchical metadata layout sized to the active working set rather than the worst-case address space. Built on vLLM with three representative sparse attention algorithms, SPIN delivers 1.66-5.66x higher end-to-end throughput and 7-9x lower TTFT than vLLM, and reduces TPOT by up to 58% over the original sparse-attention implementations.

翻译：长上下文大语言模型服务面临的核心瓶颈在于，随着键值缓存不断增长，注意力计算成本急剧上升。动态稀疏注意力通过仅访问每步解码中少量、与查询相关的键值状态子集，并将键值存储扩展至CPU内存，有望缓解这一问题。然而在实践中，这些算法层面的效率提升很少能转化为端到端系统级增益，因为稀疏方法通常在不同粒度下运行，需依赖针对特定算法的临时实现。同时，分层键值存储引入新的系统瓶颈：跨GPU-CPU边界检索细粒度、不规则的键值子集，极易抵消稀疏化带来的收益。我们提出SPIN——一种稀疏注意力感知推理框架，通过三种技术协同设计执行流水线与分层键值存储：（1）统一划分抽象，将不同稀疏粒度映射至共享的基于页面的键值存储基板；（2）局部性感知的键值缓存管理器，动态分配请求级HBM预算，并采用GPU友好的分桶LRU策略减少PCIe往返次数；（3）两级分层元数据布局，其规模依据活跃工作集而非最坏情况地址空间确定。基于vLLM集成三种代表性稀疏注意力算法的实验表明，SPIN相比vLLM实现1.66-5.66倍端到端吞吐量提升，首令牌延迟降低7-9倍，且相较于原始稀疏注意力实现，单令牌生成时延最高降低58%。