DistServe improves the performance of large language models (LLMs) serving by disaggregating the prefill and decoding computation. Existing LLM serving systems colocate the two phases and batch the computation of prefill and decoding across all users and requests. We find that this strategy not only leads to strong prefill-decoding interferences but also couples the resource allocation and parallelism plans for both phases. LLM applications often emphasize individual latency for each phase: time to first token (TTFT) for the prefill phase and time per output token (TPOT) of each request for the decoding phase. In the presence of stringent latency requirements, existing systems have to prioritize one latency over the other, or over-provision compute resources to meet both. DistServe assigns prefill and decoding computation to different GPUs, hence eliminating prefill-decoding interferences. Given the application's TTFT and TPOT requirements, DistServe co-optimizes the resource allocation and parallelism strategy tailored for each phase. DistServe also places the two phases according to the serving cluster's bandwidth to minimize the communication caused by disaggregation. As a result, DistServe significantly improves LLM serving performance in terms of the maximum rate that can be served within both TTFT and TPOT constraints on each GPU. Our evaluations show that on various popular LLMs, applications, and latency requirements, DistServe can serve 4.48x more requests or 10.2x tighter SLO, compared to state-of-the-art systems, while staying within latency constraints for > 90% of requests.

翻译：DistServe通过分离预填充（Prefill）与解码（Decoding）计算，提升了大型语言模型（LLMs）的服务性能。现有LLM服务系统将两个阶段合并处理，并对所有用户和请求的预填充、解码计算进行批量化。我们发现，这种策略不仅导致预填充与解码之间存在强干扰，还会将两个阶段的资源分配与并行化方案耦合在一起。LLM应用通常对每个阶段有各自的延迟要求：预填充阶段的首词延迟（TTFT）和解码阶段的每个请求输出令牌时间（TPOT）。在严格的延迟约束下，现有系统不得不优先满足某一延迟指标，或过度配置计算资源以同时满足两者。DistServe将预填充和解码计算分配到不同的GPU上，从而消除了预填充-解码干扰。根据应用对TTFT和TPOT的需求，DistServe协同优化每个阶段特化的资源分配与并行化策略。此外，DistServe根据服务集群的带宽放置两个阶段，以最小化解耦导致的通信开销。最终，DistServe在满足TTFT和TPOT约束的条件下，显著提升了LLM服务在每块GPU上可服务的最大速率。评估表明，在多种主流LLM、应用和延迟要求下，与最先进系统相比，DistServe可多服务4.48倍请求或实现10.2倍更严格的SLO，且超过90%的请求保持在延迟约束内。