The efficient deployment of large language models (LLMs) in online settings requires optimizing inference performance under stringent latency constraints, particularly the time-to-first-token (TTFT) and time-per-output-token (TPOT). This paper focuses on the query scheduling problem for LLM inference with prefix reuse, a technique that leverages shared prefixes across queries to reduce computational overhead. Our work reveals previously unknown limitations of the existing first-come-first-serve (FCFS) and longest-prefix-match (LPM) scheduling strategies with respect to satisfying latency constraints. We present a formal theoretical framework for LLM query scheduling under RadixAttention, a prefix reuse mechanism that stores and reuses intermediate representations in a radix tree structure. Our analysis establishes the NP-hardness of the scheduling problem with prefix reuse under TTFT constraints and proposes a novel scheduling algorithm, $k$-LPM, which generalizes existing methods by balancing prefix reuse and fairness in query processing. Theoretical guarantees demonstrate that $k$-LPM achieves improved TTFT performance under realistic traffic patterns captured by a data generative model. Empirical evaluations in a realistic serving setting validates our findings, showing significant reductions in P99 TTFT compared to baseline methods.

翻译：在线场景下高效部署大语言模型（LLMs）需要在严格的延迟约束下优化推理性能，特别是首令牌生成时间（TTFT）与每输出令牌时间（TPOT）。本文聚焦于支持前缀复用的LLM推理查询调度问题，该技术通过跨查询共享前缀来降低计算开销。我们的工作揭示了现有先到先服务（FCFS）与最长前缀匹配（LPM）调度策略在满足延迟约束方面存在此前未知的局限性。我们提出了RadixAttention（一种在基数树结构中存储并复用中间表示的前缀复用机制）下LLM查询调度的形式化理论框架。分析证明了在TTFT约束下带前缀复用的调度问题具有NP难度，并提出了一种新型调度算法$k$-LPM，该算法通过平衡查询处理中的前缀复用与公平性来泛化现有方法。理论保证表明，在数据生成模型捕获的实际流量模式下，$k$-LPM能够实现更优的TTFT性能。实际部署环境中的实证评估验证了我们的结论，相较于基线方法，该算法在P99 TTFT指标上实现了显著降低。