Large language models (LLMs), especially Generative Pretrained Transformer (GPT) models, have significantly advanced in the industry in recent years. However, these models' broader development faces considerable challenges due to high operational and deployment costs. This has led to active research in improving the hardware efficiency of LLMs. Yet, the characteristics of real-world LLM workloads are often overlooked in current optimizations of LLM serving systems. In this work, we find that the absence of reliable workload data for evaluating LLM serving systems impacts the quality of service (QoS) and reliability in industrial deployments. This paper introduces the first real-world trace dataset of LLM serving workloads, detailing user, system, and LLM behaviors. We analyze this trace, highlighting burstiness, request and response distributions, and focusing on the reliability of GPT services. Based on this, we have developed a benchmark suite that reflects our dataset's workload patterns, enabling performance evaluation of serving systems. This suite captures the core patterns of workload distributions, allowing for precise scaling of the workload dataset to match system sizes. Our evaluation uncovers a previously unrecognized vulnerability of LLM serving systems to short-term burstiness, particularly in common workload scenarios. We observe that GPU memory limitations, caused by the fluctuating nature of burstiness, lead to significant performance degradation in existing LLM serving systems. Beyond benchmarking, understanding these patterns is valuable for optimizing LLM workload management, enabling elastic hardware resource adjustments to varying workloads. We will make the dataset and benchmark suite publicly available to encourage further research.

翻译：大型语言模型(LLM)，特别是生成式预训练Transformer(GPT)模型，近年来在工业界取得了显著进展。然而，由于高昂的运营和部署成本，这些模型的更广泛发展面临着巨大挑战。这促使人们积极研究提高LLM的硬件效率。然而，当前LLM服务系统的优化往往忽视了真实世界LLM工作负载的特性。本研究发现，缺乏用于评估LLM服务系统的可靠工作负载数据会影响工业部署中的服务质量(QoS)和可靠性。本文首次介绍了LLM服务工作负载的真实世界追踪数据集，详细描述了用户、系统和LLM的行为。我们分析了该追踪，强调了突发性、请求和响应分布，并重点关注GPT服务的可靠性。基于此，我们开发了一个基准测试套件，反映了我们数据集中的工作负载模式，从而能够对服务系统进行性能评估。该套件捕捉了工作负载分布的核心模式，允许精确缩放工作负载数据集以匹配系统规模。我们的评估揭示了LLM服务系统在短期突发性方面此前未被认识到的脆弱性，尤其是在常见的工作负载场景中。我们观察到，由突发性的波动特性导致的GPU内存限制，使得现有LLM服务系统的性能显著下降。除了基准测试外，理解这些模式对于优化LLM工作负载管理也很有价值，能够实现弹性硬件资源对变化工作负载的调整。我们将公开数据集和基准测试套件，以鼓励进一步研究。