The rapid proliferation of Large Language Models (LLMs) has been a driving force in the growth of cloud-based LLM services, which are now integral to advancing AI applications. However, the dynamic auto-regressive nature of LLM service, along with the need to support exceptionally long context lengths, demands the flexible allocation and release of substantial resources. This presents considerable challenges in designing cloud-based LLM service systems, where inefficient management can lead to performance degradation or resource wastage. In response to these challenges, this work introduces DistAttention, a novel distributed attention algorithm that segments the KV Cache into smaller, manageable units, enabling distributed processing and storage of the attention module. Based on that, we propose DistKV-LLM, a distributed LLM serving system that dynamically manages KV Cache and effectively orchestrates all accessible GPU and CPU memories spanning across the data center. This ensures a high-performance LLM service on the cloud, adaptable to a broad range of context lengths. Validated in a cloud environment with 32 NVIDIA A100 GPUs in configurations from 2 to 32 instances, our system exhibited 1.03-2.4x end-to-end throughput improvements and supported context lengths 2-19x longer than current state-of-the-art LLM service systems, as evidenced by extensive testing across 18 datasets with context lengths up to 1,900K.

翻译：摘要：大语言模型（LLM）的快速普及推动了基于云的LLM服务发展，这类服务已成为促进AI应用进步的核心要素。然而，LLM服务固有的动态自回归特性以及对超长上下文长度的支持需求，要求对大量资源进行灵活分配与释放，这给基于云的LLM服务系统设计带来了巨大挑战——资源管理低效可能导致性能下降或资源浪费。针对这些问题，本文提出DistAttention算法，一种新型分布式注意力机制，将KV缓存拆分为更易管理的单元，实现注意力模块的分布式处理与存储。在此基础上，我们构建了DistKV-LLM分布式LLM服务系统，该系统可动态管理KV缓存，并有效协调数据中心内所有可访问的GPU与CPU内存资源。该方案确保云环境下的高性能LLM服务能够适应广泛的上下文长度需求。在包含32块NVIDIA A100 GPU的云环境中，通过2至32实例配置的实测验证，本系统在端到端吞吐量上提升1.03-2.4倍，支持的上下文长度较当前最先进的LLM服务系统延长2-19倍——这一结论基于对18个数据集（上下文长度最高达1,900K）的广泛测试。