DistServe improves the performance of large language models (LLMs) serving by disaggregating the prefill and decoding computation. Existing LLM serving systems colocate the two phases and batch the computation of prefill and decoding across all users and requests. We find that this strategy not only leads to strong prefill-decoding interferences but also couples the resource allocation and parallelism plans for both phases. LLM applications often emphasize individual latency for each phase: time to first token (TTFT) for the prefill phase and time per output token (TPOT) of each request for the decoding phase. In the presence of stringent latency requirements, existing systems have to prioritize one latency over the other, or over-provision compute resources to meet both. DistServe assigns prefill and decoding computation to different GPUs, hence eliminating prefill-decoding interferences. Given the application's TTFT and TPOT requirements, DistServe co-optimizes the resource allocation and parallelism strategy tailored for each phase. DistServe also places the two phases according to the serving cluster's bandwidth to minimize the communication caused by disaggregation. As a result, DistServe significantly improves LLM serving performance in terms of the maximum rate that can be served within both TTFT and TPOT constraints on each GPU. Our evaluations show that on various popular LLMs, applications, and latency requirements, DistServe can serve 4.48x more requests or 10.2x tighter SLO, compared to state-of-the-art systems, while staying within latency constraints for > 90% of requests.

翻译：DistServe通过分离预填充（Prefill）与解码（Decoding）计算，提升了大语言模型（LLM）的服务性能。现有LLM服务系统将这两个阶段合并处理，并在所有用户和请求中批量执行预填充与解码计算。我们发现，这种策略不仅导致强烈的预填充-解码相互干扰，还耦合了两个阶段的资源分配与并行化方案。LLM应用通常对每个阶段的独立延迟有明确要求：预填充阶段的首令牌时间（TTFT）和解码阶段的每输出令牌时间（TPOT）。在严格的延迟约束下，现有系统不得不优先满足某一延迟指标，或过度配置计算资源以同时满足两者。DistServe将预填充和解码计算分配到不同GPU上，从而消除了预填充-解码干扰。根据应用的TTFT和TPOT需求，DistServe为每个阶段协同优化资源分配与并行化策略。此外，DistServe还根据服务集群的带宽对两个阶段进行放置，以最小化解聚带来的通信开销。最终，DistServe显著提升了LLM服务性能，在每GPU上同时满足TTFT和TPOT约束的情况下，最大化可服务的请求速率。我们的评估表明，在多种主流LLM、应用场景及延迟约束条件下，与现有最先进系统相比，DistServe可处理4.48倍以上的请求，或在>90%的请求满足延迟约束的前提下，实现10.2倍更严格的SLO。