Large language models (LLMs) have demonstrated remarkable performance, and organizations are racing to serve LLMs of varying sizes as endpoints for use-cases like chat, programming and search. However, efficiently serving multiple LLMs poses significant challenges for existing approaches due to varying popularity of LLMs. In the paper, we present MuxServe, a flexible spatial-temporal multiplexing system for efficient multiple LLM serving. The key insight behind is to colocate LLMs considering their popularity to multiplex memory resources, and leverage the characteristics of prefill and decoding phases to separate and flexibly colocate them to multiplex computation resources. MuxServe formally formulates the multiplexing problem, and proposes a novel placement algorithm and adaptive batch scheduling strategy to identify optimal colocations and maximize utilization. MuxServe designs a unified resource manager to enable flexible and efficient multiplexing. Evaluation results show that MuxServe can achieves up to $1.8\times$ higher throughput or processes $2.9\times$ more requests within $99\%$ SLO attainment.

翻译：大型语言模型（LLMs）展现出卓越性能，各机构正竞相将不同规模的LLM部署为服务端点，用于聊天、编程和搜索等场景。然而，由于LLM流行度的差异，现有方法在高效服务多个LLM时面临严峻挑战。本文提出MuxServe——一种灵活的时空复用系统，用于实现多LLM的高效服务。其核心思想在于：根据LLM的流行度进行协同部署以复用内存资源，并利用预填充与解码阶段的特性实现计算资源的分离与灵活协同复用。MuxServe正式形式化定义了复用问题，通过创新的放置算法与自适应批调度策略，识别最优协同部署方案并最大化资源利用率。该框架设计了统一资源管理器以支持灵活高效的复用。评估结果表明，MuxServe可实现最高1.8倍的吞吐量提升，或在99%服务等级协议（SLO）达标率下处理2.9倍以上的请求。