Large language models (LLMs) have demonstrated remarkable performance, and organizations are racing to serve LLMs of varying sizes as endpoints for use-cases like chat, programming and search. However, efficiently serving multiple LLMs poses significant challenges for existing approaches due to varying popularity of LLMs. In the paper, we present MuxServe, a flexible spatial-temporal multiplexing system for efficient multiple LLM serving. The key insight behind is to colocate LLMs considering their popularity to multiplex memory resources, and leverage the characteristics of prefill and decoding phases to separate and flexibly colocate them to multiplex computation resources. MuxServe formally formulates the multiplexing problem, and proposes a novel placement algorithm and adaptive batch scheduling strategy to identify optimal colocations and maximize utilization. MuxServe designs a unified resource manager to enable flexible and efficient multiplexing. Evaluation results show that MuxServe can achieves up to $1.8\times$ higher throughput or processes $2.9\times$ more requests within $99\%$ SLO attainment. The code is available at: \url{https://github.com/hao-ai-lab/MuxServe}.

翻译：大型语言模型（LLM）已展现出卓越的性能，各机构正竞相部署不同规模的LLM作为终端服务，以支持聊天、编程和搜索等应用场景。然而，由于不同LLM的受欢迎程度存在差异，高效服务多个LLM对现有方法提出了重大挑战。本文提出MuxServe，一种灵活的时空复用系统，用于高效服务多个LLM。其核心思想在于：根据LLM的流行度进行协同放置以实现内存资源复用，并利用预填充和解码阶段的特性对两者进行分离与灵活协同放置以实现计算资源复用。MuxServe将复用问题形式化建模，并提出一种新颖的放置算法与自适应批调度策略，以确定最优协同放置方案并最大化资源利用率。系统设计了统一的资源管理器以实现灵活高效的复用。评估结果表明，在满足99%服务等级目标的前提下，MuxServe可实现高达$1.8\times$的吞吐量提升，或处理$2.9\times$更多的请求。代码已开源：\url{https://github.com/hao-ai-lab/MuxServe}。