Fine-tuning large language models (LLMs) for downstream tasks can greatly improve model quality, however serving many different fine-tuned LLMs concurrently for users in multi-tenant environments is challenging. Dedicating GPU memory for each model is prohibitively expensive and naively swapping large model weights in and out of GPU memory is slow. Our key insight is that fine-tuned models can be quickly swapped in and out of GPU memory by extracting and compressing the delta between each model and its pre-trained base model. We propose DeltaZip, an LLM serving system that efficiently serves multiple full-parameter fine-tuned models concurrently by aggressively compressing model deltas by a factor of $6\times$ to $8\times$ while maintaining high model quality. DeltaZip increases serving throughput by $1.5\times$ to $3\times$ and improves SLO attainment compared to a vanilla HuggingFace serving system.

翻译：针对下游任务微调大型语言模型能显著提升模型质量，但在多租户环境中为用户同时部署多个不同微调模型面临挑战。为每个模型独占GPU内存代价高昂，且简单地将大模型权重移入/移出GPU内存速度缓慢。我们的关键发现是：通过提取并压缩每个微调模型与其预训练基座模型之间的参数增量，可实现模型的快速换入/换出。为此提出DeltaZip系统，通过将模型增量压缩至$6\times$到$8\times$且保持高质量，高效地同时服务多个全参数微调模型。与原生HuggingFace服务系统相比，DeltaZip将服务吞吐量提升$1.5\times$至$3\times$，并提升了SLO达标率。