The rapid growth of large language model (LLM) deployments has made cost-efficient serving systems essential. Recent efforts to enhance system cost-efficiency adopt two main perspectives: (i) An algorithmic perspective that exploits heterogeneous model capabilities to route simpler queries to lower-cost models and complex queries to higher-cost models (i.e., heterogeneous query routing); and (ii) a systems perspective that utilizes heterogeneous GPU resources as cost-effective alternatives to homogeneous high-end GPUs (i.e., heterogeneous model deployment). However, algorithm-system co-design for cost-efficient LLM serving necessitates sophisticated management: (i) Determining optimal query routing strategies under latency and quality requirements, (ii) configuring model deployment across heterogeneous GPUs with appropriate resource allocation and parallelism strategies, and (iii) co-optimizing routing and deployment decisions to maximize overall system performance. To address these challenges, we present BOute, a quality-aware scheduling system that jointly exploits heterogeneous model and GPU capabilities for cost-efficient LLM serving. BOute employs a multi-objective Bayesian optimization (MOBO) framework to co-optimize the routing strategy and model deployment, thereby maximizing the cost-efficiency of the serving system while guaranteeing response quality. Evaluation results demonstrate that BOute outperforms state-of-the-art LLM serving systems by up to 157% and 59% on average under identical cost budgets and quality requirements, or reducing serving costs by 15%-61% (38% on average) while maintaining the same performance targets, validating its effectiveness in achieving cost-efficient LLM serving.

翻译：大型语言模型（LLM）部署的快速增长使得成本高效的服务系统变得至关重要。近期提升系统成本效率的研究主要从两个视角展开：（i）算法视角：利用异构模型能力，将简单查询路由至低成本模型，复杂查询路由至高成本模型（即异构查询路由）；（ii）系统视角：利用异构GPU资源作为同构高端GPU的成本效益替代方案（即异构模型部署）。然而，面向成本高效LLM服务的算法-系统协同设计需要精细化管理：（i）在延迟与质量约束下确定最优查询路由策略；（ii）在异构GPU上配置模型部署，包括合理的资源分配与并行策略；（iii）协同优化路由与部署决策以最大化整体系统性能。为应对这些挑战，我们提出BOute——一个质量感知的调度系统，通过联合利用异构模型与GPU能力实现成本高效的LLM服务。BOute采用多目标贝叶斯优化（MOBO）框架，协同优化路由策略与模型部署，从而在保障响应质量的前提下最大化服务系统的成本效益。评估结果表明，在相同成本预算与质量要求下，BOute相比现有最优LLM服务系统性能提升最高达157%，平均提升59%；或在保持相同性能目标下将服务成本降低15%-61%（平均降低38%），验证了其在实现成本高效LLM服务方面的有效性。