With the rapid evolution of Large Language Models (LLMs), multi-round workflows, such as autonomous agents and iterative retrieval, have become increasingly prevalent. However, this raises hurdles for serving LLMs under prefill-decode (PD) disaggregation, a widely adopted paradigm that separates the compute-bound prefill phase and memory-bound decode phase onto individual resources. Specifically, existing systems overlook the interleaved prefill-decode workload pattern in multi-round inference, leading to sub-optimal handling of the incremental prefill workloads and model deployment for the two phases. In this work, we present AMPD, a brand new disaggregated serving framework for multi-round LLM inference. The core of AMPD is to coordinate the prefill workloads based on real-time workloads by adaptively determining where to carry out these workloads and how they are scheduled, in order to maximize service level objective (SLO) attainment. In addition, we tailor a planning algorithm for our scenario, facilitating the deduction of optimal resource allocation and parallel strategies for the two phases. Empirical results demonstrate that AMPD substantially improves SLO attainment compared to state-of-the-art baselines.

翻译：随着大语言模型（LLM）的快速发展，多轮工作流（例如自主智能体和迭代检索）已变得越来越普遍。然而，这为在预填充-解码（PD）解耦范式下服务LLM带来了挑战。该范式是一种广泛采用的架构，它将计算密集的预填充阶段和内存密集的解码阶段分离到独立的计算资源上。具体而言，现有系统忽视了多轮推理中交织的预填充-解码工作负载模式，导致对增量式预填充工作负载的处理以及两个阶段的模型部署均未达到最优。在本工作中，我们提出了AMPD，一个全新的面向多轮LLM推理的解耦式服务框架。AMPD的核心在于根据实时工作负载协调预填充任务，通过自适应地决定在何处执行这些任务以及如何调度它们，以最大化服务等级目标（SLO）达成率。此外，我们为本场景定制了一个规划算法，以辅助推导出两个阶段的最优资源分配与并行策略。实验结果表明，与最先进的基线方法相比，AMPD显著提升了SLO达成率。