As modern LLMs support thousands to millions of tokens, KV caches grow to hundreds of gigabytes, stressing memory capacity and bandwidth. Existing solutions, such as KV cache pruning and offloading, alleviate these but underutilize hardware by relying solely on either GPU or CPU for attention computing, and considering yet limited CPU local memory for KV cache storage. We propose HybridGen, an efficient hybrid attention framework for long-context LLM inference. HybridGen enables CPU-GPU collaborative attention on systems with expanded tiered memory (e.g., CXL memory), addressing three key challenges: (1) multi-dimensional attention dependencies, (2) intensifying CPU-GPU load imbalance with longer sequences, and (3) NUMA penalty of tiered memories. HybridGen tackles these by introducing attention logit parallelism, a feedback-driven scheduler, and semantic-aware KV cache mapping. Experiments with three LLM models with eleven different sizes on three GPU platforms with a CXL-expanded memory show that HybridGen outperforms six state-of-the-art KV cache management methods by 1.41x--3.2x on average while maintaining superior accuracy.

翻译：随着现代大语言模型支持数千至数百万个令牌，KV缓存增长至数百吉字节，对内存容量和带宽构成了严峻压力。现有解决方案（如KV缓存剪枝与卸载）虽能缓解此类问题，但仅依赖GPU或CPU进行注意力计算，且对KV缓存存储的CPU本地内存利用有限，导致硬件利用率不足。我们提出HybridGen——一种面向长上下文LLM推理的高效混合注意力框架。HybridGen可在扩展分层内存（如CXL内存）的系统上实现CPU-GPU协作式注意力计算，并解决三个关键挑战：（1）多维注意力依赖关系；（2）随序列增长加剧的CPU-GPU负载不均衡；（3）分层内存导致的NUMA开销。为此，HybridGen引入了注意力对数并行化、反馈驱动调度器及语义感知的KV缓存映射。实验采用三种LLM模型（涵盖十一种不同规模）、三个GPU平台及CXL扩展内存，结果表明HybridGen在保持卓越精度的同时，性能较六种最先进的KV缓存管理方法平均提升1.41倍至3.2倍。