Low-rank adaptation (LoRA) has become an important and popular method to adapt pre-trained models to specific domains. We present Punica, a system to serve multiple LoRA models in a shared GPU cluster. Punica contains a new CUDA kernel design that allows batching of GPU operations for different LoRA models. This allows a GPU to hold only a single copy of the underlying pre-trained model when serving multiple, different LoRA models, significantly enhancing GPU efficiency in terms of both memory and computation. Our scheduler consolidates multi-tenant LoRA serving workloads in a shared GPU cluster. With a fixed-sized GPU cluster, our evaluations show that Punica achieves 12x higher throughput in serving multiple LoRA models compared to state-of-the-art LLM serving systems while only adding 2ms latency per token. Punica is open source at https://github.com/punica-ai/punica .

翻译：低秩适配（LoRA）已成为将预训练模型适配到特定领域的重要且流行方法。我们提出Punica——一个在共享GPU集群中服务多个LoRA模型的系统。Punica包含一种新的CUDA核设计，允许对不同LoRA模型的GPU操作进行批处理。这使得在服务多个不同LoRA模型时，GPU仅需保留基础预训练模型的单一副本，从而在内存和计算两方面显著提升GPU效率。我们的调度器整合了共享GPU集群中的多租户LoRA服务工作负载。在固定规模的GPU集群上，评估显示，与最先进的LLM服务系统相比，Punica在服务多个LoRA模型时吞吐量提升12倍，且每个token仅增加2毫秒延迟。Punica已在https://github.com/punica-ai/punica开源。