Cost of serving large language models (LLM) is high, but the expensive and scarce GPUs are poorly efficient when generating tokens sequentially, unless the batch of sequences is enlarged. However, the batch size is limited by some constantly reused intermediate results, namely KV-Cache. They occupy too much memory to fit more sequences into a GPU simultaneously. While they could be offloaded to host memory, the CPU-GPU bandwidth is an inevitable bottleneck. We find a way to decompose the transformer models into two parts of different characteristics, one of which includes the memory-bound KV-Cache accessing. Our key insight is that the aggregated memory capacity, bandwidth, and computing power of CPUs across multiple nodes is an efficient option to process this part. Performance improvement comes from reduced data transmission overhead and boosted GPU throughput to process the other model part. Moreover, we address efficiency challenges brought by heterogeneity at both temporal and inter-device scopes using scheduling and performance modeling techniques. Evaluation results show that our system achieves 1.88x - 5.04x the throughput of vLLM when serving modern LLMs with the same GPU.

翻译：服务大语言模型（LLM）的成本高昂，但昂贵且稀缺的GPU在串行生成token时效率低下，除非扩大序列批次。然而，批次大小受限于部分不断重用的中间结果（即KV-Cache），这些结果占用过多内存，导致无法在同一GPU上容纳更多序列。虽然可将其卸载至主机内存，但CPU-GPU带宽成为不可避免的瓶颈。我们发现可将Transformer模型分解为两个特性不同的部分，其中一部分包含内存受限的KV-Cache访问。核心洞察在于：跨多个节点的CPU聚合内存容量、带宽与计算能力，是处理该部分的高效方案。性能提升源于减少数据传输开销并提升处理另一模型部分的GPU吞吐量。此外，我们通过调度与性能建模技术，解决了跨时间和跨设备异构性带来的效率挑战。评估结果表明，在服务现代LLM时，本系统使用相同GPU可达到vLLM吞吐量的1.88倍至5.04倍。