Large language models (LLMs) are increasingly used for complex tasks requiring multiple chained generation calls, advanced prompting techniques, control flow, and interaction with external environments. However, efficient systems for programming and executing these applications are lacking. To bridge this gap, we introduce SGLang, a Structured Generation Language for LLMs. SGLang is designed for the efficient programming of LLMs and incorporates primitives for common LLM programming patterns. We have implemented SGLang as a domain-specific language embedded in Python, and we developed an interpreter, a compiler, and a high-performance runtime for SGLang. These components work together to enable optimizations such as parallelism, batching, caching, sharing, and other compilation techniques. Additionally, we propose RadixAttention, a novel technique that maintains a Least Recently Used (LRU) cache of the Key-Value (KV) cache for all requests in a radix tree, enabling automatic KV cache reuse across multiple generation calls at runtime. SGLang simplifies the writing of LLM programs and boosts execution efficiency. Our experiments demonstrate that SGLang can speed up common LLM tasks by up to 5x, while reducing code complexity and enhancing control.

翻译：大语言模型（LLMs）正越来越多地用于需要多步链式生成、高级提示技术、控制流以及与外部环境交互的复杂任务。然而，目前缺乏用于编程和执行这些应用的高效系统。为填补这一空白，我们提出SGLang——一种面向LLMs的结构化生成语言。SGLang专为高效编程LLMs而设计，并包含常见LLM编程模式的原语。我们已将SGLang实现为一种嵌入Python的领域特定语言，并为其开发了解释器、编译器及高性能运行时环境。这些组件协同工作，支持并行化、批处理、缓存、共享及其他编译优化技术。此外，我们还提出了RadixAttention——一种创新技术，通过基数树维护所有请求的键值缓存（KV Cache）的最近最少使用（LRU）缓存，从而实现运行时跨多次生成调用的自动KV缓存复用。SGLang简化了LLM程序的编写过程并提升了执行效率。实验表明，SGLang可将常见LLM任务的速度提升最多5倍，同时降低代码复杂度并增强控制力。