In production environments, large language model (LLM) serving is required to meet stringent service-level objectives (SLOs) amid highly variable request patterns. In practice, request lengths follow a long-tail distribution, which gives rise to head-of-line blocking on the prefill side and underutilization caused by stragglers on the decode side in disaggregated serving architectures. Current systems, which adopt first-come-first-served (FCFS) scheduling for prefill and continuous batching for decode, lack the ability to adapt to this imbalance, resulting in compromised SLO attainment and reduced throughput. To address these challenges, we propose Kairos, an SLO-aware scheduling system equipped with two complementary mechanisms. On the prefill side, Kairos employs urgency-based priority scheduling: it predicts prefill completion times and dynamically selects requests to maximize the attainment of time-to-first-token (TTFT) SLOs. On the decode side, Kairos introduces slack-guided adaptive batching, which leverages the gap between per-step decode time and the time-per-output-token (TPOT) SLO to greedily pack short requests. This approach maximizes throughput while strictly adhering to SLO requirements. We implement Kairos and conduct evaluations using an online serving dataset and a state-of-the-art LLM. Experimental results demonstrate that, compared with state-of-the-art baselines, Kairos improves TTFT SLO attainment by up to 23.9\%, TPOT SLO attainment by up to 27.1\%, end-to-end SLO attainment by up to 33.8\%, and decode throughput by up to 19.3\%.

翻译：在生产环境中，大语言模型（LLM）服务需要在高度变化的请求模式下满足严格的服务等级目标（SLO）。实际应用中，请求长度呈现长尾分布，这导致在解聚式服务架构中预填充阶段出现队头阻塞，解码阶段因掉队者导致利用率不足。当前采用先来先服务（FCFS）调度预填充和连续批处理解码的系统，无法适应这种不均衡，导致SLO达标率受损和吞吐量下降。针对这些挑战，我们提出Kairos——配备两种互补机制的SLO感知调度系统。在预填充侧，Kairos采用基于紧迫性的优先级调度：通过预测预填充完成时间，动态选择请求以最大化首个令牌生成时间（TTFT）的SLO达标率。在解码侧，Kairos引入松弛引导的自适应批处理，利用每步解码时间与每输出令牌时间（TPOT）SLO之间的间隙，贪婪地打包短请求。该方法在严格遵循SLO要求的同时最大化吞吐量。我们基于在线服务数据集和最先进LLM实现了Kairos系统并进行评估。实验结果表明，与最先进基线相比，Kairos将TTFT SLO达标率提升最高23.9%，TPOT SLO达标率提升最高27.1%，端到端SLO达标率提升最高33.8%，解码吞吐量提升最高19.3%。