Serving Large Language Models (LLMs) under mixed workloads--short, latency-sensitive interactive queries alongside long, throughput-oriented batch requests--poses a fundamental scheduling challenge. Standard First-Come, First-Served (FCFS) policies suffer from severe head-of-line blocking, leading to high tail latency and underutilized hardware. We introduce EWSJF (Effective Workload-based Shortest Job First), an adaptive request-level scheduler that learns workload structure in real time to jointly improve fairness and throughput. EWSJF operates upstream of execution-level schedulers and integrates four components: (1) Refine-and-Prune, an unsupervised partitioning algorithm that discovers performance-homogeneous request groups; (2) Dynamic Queue Routing for assigning requests to these groups; (3) Density-Weighted Scoring, a context-aware prioritization function balancing urgency and fairness; and (4) Bayesian Meta-Optimization, which continuously tunes scoring and partitioning parameters based on live performance feedback. Implemented in vLLM, EWSJF improves end-to-end throughput by over 30% and reduces average Time-To-First-Token for short requests by up to 4x compared to FCFS. These results demonstrate that adaptive, learning-based request scheduling is a critical missing layer for efficient and responsive LLM serving. Implementation available at https://anonymous.4open.science/r/vllm_0110-32D8.

翻译：在混合工作负载（即同时存在短时延敏感的交互式查询与长吞吐导向的批处理请求）下服务大语言模型（LLMs）面临根本性的调度挑战。标准的先到先服务（FCFS）策略会遭受严重的队头阻塞，导致高尾延迟和硬件利用率不足。本文提出EWSJF（基于有效工作负载的最短作业优先），一种自适应请求级调度器，能够实时学习工作负载结构，以协同提升公平性与吞吐量。EWSJF运行在执行级调度器之上，并集成四个组件：（1）Refine-and-Prune，一种无监督分区算法，用于发现性能同质的请求组；（2）动态队列路由，用于将请求分配至这些组；（3）密度加权评分，一种平衡紧迫性与公平性的上下文感知优先级函数；（4）贝叶斯元优化，基于实时性能反馈持续调整评分与分区参数。在vLLM中实现的EWSJF相比FCFS，端到端吞吐量提升超过30%，短请求的平均首词生成时间最多降低至1/4。这些结果表明，自适应、基于学习的请求调度是实现高效、响应迅速的大语言模型服务所缺失的关键层。实现代码发布于 https://anonymous.4open.science/r/vllm_0110-32D8。