We study offline scheduling for large language model (LLM) serving under a fixed KV-cache memory budget, where requests have heterogeneous prompt (prefill) and response (decode) lengths. Prompt tokens determine initial KV usage, and each generated token increases memory by one unit. Given a backlog of n requests arriving together, we schedule mixed prefill and decode batches to minimize total end-to-end latency. We show that heterogeneity in prompt lengths makes the problem computationally intractable and that widely used heuristics such as first-come-first-served and shortest-first can be arbitrarily suboptimal. We propose Sorted-F, which repeatedly forms feasible batches using a new selection metric that balances batch size against downstream decode cost, and prove it achieves a constant-factor guarantee on total latency. We further develop practical variants -- an exact solver for small instances and fast heuristics for larger ones -- and evaluate them on a public workload spanning short conversations and long-document summarization, where they consistently reduce average latency relative to standard baselines. Our results highlight that during peak-hour tidal backlogs, greedy GPU packing or short-request prioritization can perform poorly when prompt lengths vary widely, and provide a principled, tunable framework for designing production batch schedulers and planning capacity in memory-constrained LLM serving systems.

翻译：本文研究在固定键值缓存（KV-cache）内存预算下，针对具有异构提示（预填充）和响应（解码）长度请求的大语言模型（LLM）服务的离线调度问题。提示词决定初始键值缓存使用量，每个生成词符会使内存增加一个单位。给定同时到达的n个请求积压队列，我们通过调度混合的预填充与解码批次来最小化端到端总延迟。研究表明：提示长度的异构性导致该问题在计算上难以处理，且广泛使用的先到先服务、最短优先等启发式算法可能产生任意程度的次优解。我们提出Sorted-F算法，该算法通过采用平衡批次规模与下游解码成本的新型选择度量，反复构建可行批次，并证明其能在总延迟上实现常数倍性能保证。进一步开发了实用变体——针对小规模实例的精确求解器与针对大规模实例的快速启发式算法，并在涵盖短对话与长文档摘要的公开工作负载上进行评估，结果表明这些方法相较于标准基线能持续降低平均延迟。研究结果强调：在高峰时段的海量积压请求中，当提示长度差异较大时，贪婪的GPU打包策略或短请求优先策略可能表现不佳，同时为内存受限的LLM服务系统中生产级批次调度器的设计与容量规划提供了可调节的原则性框架。