The growing usage of Large Language Models (LLMs) highlights the demands and challenges in scalable LLM inference systems, affecting deployment and development processes. On the deployment side, there is a lack of comprehensive analysis on the conditions under which a particular scheduler performs better or worse, with performance varying substantially across different schedulers, hardware, models, and workloads. Manually testing each configuration on GPUs can be prohibitively expensive. On the development side, unpredictable performance and unknown upper limits can lead to inconclusive trial-and-error processes, consuming resources on ideas that end up ineffective. To address these challenges, we introduce INFERMAX, an analytical framework that uses inference cost models to compare various schedulers, including an optimal scheduler formulated as a constraint satisfaction problem (CSP) to establish an upper bound on performance. Our framework offers in-depth analysis and raises essential questions, challenging assumptions and exploring opportunities for more efficient scheduling. Notably, our findings indicate that preempting requests can reduce GPU costs by 30% compared to avoiding preemptions at all. We believe our methods and insights will facilitate the cost-effective deployment and development of scalable, efficient inference systems and pave the way for cost-based scheduling.

翻译：大型语言模型（LLM）日益广泛的应用凸显了可扩展LLM推理系统在部署与开发过程中面临的需求与挑战。在部署层面，目前缺乏对特定调度器在何种条件下表现更优或更差的全面分析，其性能在不同调度器、硬件、模型及工作负载间存在显著差异。在GPU上手动测试每种配置的成本可能极其高昂。在开发层面，不可预测的性能与未知的性能上限可能导致试错过程难以得出结论，从而将资源消耗在最终无效的方案上。为应对这些挑战，我们提出了INFERMAX——一种基于推理成本模型的分析框架，用于比较各类调度器，其中包括一个表述为约束满足问题（CSP）的最优调度器，以确立性能上界。该框架提供深度分析并提出关键问题，挑战既有假设并探索提升调度效率的潜在途径。值得注意的是，我们的研究结果表明：相较于完全避免抢占请求，采用抢占策略可降低30%的GPU成本。我们相信，本研究所提出的方法与洞见将助力构建具备成本效益的可扩展高效推理系统，并为基于成本的调度机制开辟道路。