The growing demand for large language models (LLMs) requires serving systems to handle many concurrent requests with diverse service level objectives (SLOs). This exacerbates head-of-line (HoL) blocking during the compute-intensive prefill phase, where long-running requests monopolize resources and delay higher-priority ones, leading to widespread time-to-first-token (TTFT) SLO violations. While chunked prefill enables interruptibility, it introduces an inherent trade-off between responsiveness and throughput: reducing chunk size improves response latency but degrades computational efficiency, whereas increasing chunk size maximizes throughput but exacerbates blocking. This necessitates an adaptive preemption mechanism. However, dynamically balancing execution granularity against scheduling overheads remains a key challenge. In this paper, we propose FlowPrefill, a TTFT-goodput-optimized serving system that resolves this conflict by decoupling preemption granularity from scheduling frequency. To achieve adaptive prefill scheduling, FlowPrefill introduces two key innovations: 1) Operator-Level Preemption, which leverages operator boundaries to enable fine-grained execution interruption without the efficiency loss associated with fixed small chunking; and 2) Event-Driven Scheduling, which triggers scheduling decisions only upon request arrival or completion events, thereby supporting efficient preemption responsiveness while minimizing control-plane overhead. Evaluation on real-world production traces shows that FlowPrefill improves maximum goodput by up to 5.6$\times$ compared to state-of-the-art systems while satisfying heterogeneous SLOs.

翻译：随着大语言模型（LLM）需求的增长，服务系统需要处理大量具有多样化服务水平目标（SLO）的并发请求。这加剧了计算密集型预填充阶段的队头阻塞问题：长时运行的请求独占资源，延迟了高优先级请求的处理，导致广泛的首令牌时间服务水平目标违规。虽然分块预填充实现了可中断性，但它在响应性与吞吐量之间引入了固有的权衡：减小块大小可改善响应延迟但降低计算效率，而增大块大小虽能最大化吞吐量却会加剧阻塞。这需要一种自适应的抢占机制。然而，动态平衡执行粒度与调度开销仍是一个关键挑战。本文提出FlowPrefill，一个优化首令牌时间与有效吞吐量的服务系统，通过将抢占粒度与调度频率解耦来解决这一矛盾。为实现自适应预填充调度，FlowPrefill引入了两项关键创新：1）算子级抢占，利用算子边界实现细粒度执行中断，避免了固定小分块带来的效率损失；2）事件驱动调度，仅在请求到达或完成事件时触发调度决策，从而在最小化控制面开销的同时支持高效的抢占响应性。基于真实生产轨迹的评估表明，FlowPrefill在满足异构服务水平目标的同时，相比最先进系统将最大有效吞吐量提升了最高5.6倍。