Large language models (LLMs), especially Generative Pretrained Transformer (GPT) models, have significantly advanced in the industry in recent years. However, these models' broader development faces considerable challenges due to high operational and deployment costs. This has led to active research in improving the hardware efficiency of LLMs. Yet, the characteristics of real-world LLM workloads are often overlooked in current optimizations of LLM serving systems. In this work, the absence of reliable workload data for evaluating LLM serving systems impacts the quality of service (QoS) and reliability in industrial deployments. This paper introduces the first real-world trace dataset of LLM serving workloads, detailing user, system, and LLM behaviors. We analyze this trace, highlighting burstiness, request and response distributions, and focusing on the reliability of GPT services. Based on this, we have developed a benchmark suite that reflects our dataset's workload patterns, enabling performance evaluation of serving systems. This suite captures the core patterns of workload distributions, allowing for precise scaling of the workload dataset to match system sizes. Our evaluation uncovers a previously unrecognized vulnerability of LLM serving systems to short-term burstiness, particularly in common workload scenarios. We observe that GPU memory limitations, caused by the fluctuating nature of burstiness, lead to significant performance degradation in existing LLM serving systems. Beyond benchmarking, understanding these patterns is valuable for optimizing LLM workload management, enabling elastic hardware resource adjustments to varying workloads. To encourage further research, we have made the dataset and benchmark suite publicly available at https://github.com/HPMLL/BurstGPT.

翻译：大型语言模型（LLM），尤其是生成式预训练Transformer（GPT）模型，近年来在工业界取得了显著进展。然而，由于高昂的运营和部署成本，这些模型的广泛发展仍面临巨大挑战。这促使研究人员积极寻求提升LLM硬件效率的方法。然而，在当前对LLM服务系统的优化中，真实世界LLM工作负载的特性往往被忽视。本研究指出，缺乏评估LLM服务系统的可靠工作负载数据，会直接影响工业部署中的服务质量（QoS）和可靠性。本文首次提出LLM服务工作负载的真实轨迹数据集，详细描述了用户、系统及LLM的行为特征。我们对该轨迹进行了分析，重点揭示了突发性、请求与响应分布，并聚焦于GPT服务的可靠性。基于此，我们开发了一套反映数据集工作负载模式的基准测试套件，用于评估服务系统的性能。该套件捕捉了工作负载分布的核心模式，使得工作负载数据集能够精确缩放以匹配系统规模。我们的评估揭示了一个此前未被认知的弱点：LLM服务系统极易受到短期突发性的影响，尤其是在常见工作负载场景中。我们观察到，由突发性波动特性导致的GPU内存限制，会显著降低现有LLM服务系统的性能。除基准测试外，理解这些模式有助于优化LLM工作负载管理，实现硬件资源对动态工作负载的弹性调整。为促进进一步研究，我们已将数据集和基准测试套件公开于https://github.com/HPMLL/BurstGPT。