Multi-model LLM routing has emerged as an effective approach for reducing serving cost and latency while maintaining output quality by assigning each prompt to an appropriate model. However, prior routing methods typically assume that each model has a fixed latency. In real deployments, this assumption is inaccurate: multiple models often share limited GPU resources, and a model's latency depends strongly on both its allocated resources and the request load induced by the routing policy. Consequently, routing and resource allocation are tightly coupled. In this work, we study joint resource allocation and routing for latency-aware multi-model LLM serving in GPU clusters. Given a set of deployed models and a latency service-level objective (SLO), we seek a system setup and routing policy that maximize overall output quality while satisfying the latency target. We formalize this problem as a constrained joint optimization over deployment setup and routing fractions, and propose RouterWise, which combines a dual-price formulation for score-maximizing routing with setup-specific latency models derived from system profiling. RouterWise searches over feasible system setups and, for each fixed setup, computes the best routing policy under the latency target. Our results show that even on the same GPU cluster, achievable output-quality score can vary by up to 87% across retained setups, highlighting that resource allocation is a key determinant of routing performance.

翻译：多模型LLM路由通过为每个提示分配适当的模型，已成为在保持输出质量的同时降低服务成本和延迟的有效方法。然而，现有的路由方法通常假设每个模型具有固定的延迟。在实际部署中，这一假设并不准确：多个模型通常共享有限的GPU资源，而模型的延迟强烈依赖于其分配的资源以及由路由策略引起的请求负载。因此，路由与资源分配紧密耦合。在本研究中，我们探讨了GPU集群中面向延迟感知的多模型LLM服务的联合资源分配与路由问题。给定一组部署模型和延迟服务等级目标（SLO），我们寻求一种系统设置和路由策略，以在满足延迟目标的同时最大化整体输出质量。我们将该问题形式化为一个关于部署设置和路由分数的约束联合优化问题，并提出了RouterWise，该方法结合了用于分数最大化路由的双价公式与通过系统剖析得出的特定设置延迟模型。RouterWise对所有可行的系统设置进行搜索，并针对每种固定设置，计算延迟目标下的最优路由策略。结果表明，即使在相同的GPU集群上，在不同保留设置下，可实现的输出质量分数差异可高达87%，这凸显了资源分配是路由性能的关键决定因素。