The deployment and scaling of large language models (LLMs) have become critical as they permeate various applications, demanding high-throughput and low-latency serving systems. Existing frameworks struggle to balance these requirements, especially for workloads with long prompts. This paper introduces DeepSpeed-FastGen, a system that employs Dynamic SplitFuse, a novel prompt and generation composition strategy, to deliver up to 2.3x higher effective throughput, 2x lower latency on average, and up to 3.7x lower (token-level) tail latency, compared to state-of-the-art systems like vLLM. We leverage a synergistic combination of DeepSpeed-MII and DeepSpeed-Inference to provide an efficient and easy-to-use serving system for LLMs. DeepSpeed-FastGen's advanced implementation supports a range of models and offers both non-persistent and persistent deployment options, catering to diverse user scenarios from interactive sessions to long-running applications. We present a detailed benchmarking methodology, analyze the performance through latency-throughput curves, and investigate scalability via load balancing. Our evaluations demonstrate substantial improvements in throughput and latency across various models and hardware configurations. We discuss our roadmap for future enhancements, including broader model support and new hardware backends. The DeepSpeed-FastGen code is readily available for community engagement and contribution.

翻译：随着大语言模型（LLM）在各类应用中的广泛渗透，其部署与规模化已成为关键问题，亟需高吞吐、低延迟的服务系统。现有框架难以平衡这些需求，尤其是在处理长提示词工作负载时。本文提出DeepSpeed-FastGen系统，该系统采用新颖的提示与生成组合策略Dynamic SplitFuse，相较于vLLM等最先进系统，可实现高达2.3倍的有效吞吐量提升、平均延迟降低2倍，以及高达3.7倍的（令牌级）尾部延迟降低。我们通过协同整合DeepSpeed-MII与DeepSpeed-Inference，为LLM构建了高效易用的服务系统。DeepSpeed-FastGen的先进实现支持多种模型，并提供非持久化与持久化两种部署选项，满足从交互式会话到长期运行应用的多样化用户场景。本文呈现了详细的基准测试方法，通过延迟-吞吐量曲线分析性能，并通过负载均衡研究可扩展性。实验表明，该系统在多种模型与硬件配置下均显著提升了吞吐量与延迟性能。我们讨论了未来增强路线图，包括更广泛的模型支持与新型硬件后端。DeepSpeed-FastGen代码已开放，供社区参与和贡献。