This paper presents ServerlessLLM, a locality-enhanced serverless inference system for Large Language Models (LLMs). ServerlessLLM exploits the substantial capacity and bandwidth of storage and memory devices available on GPU servers, thereby reducing costly remote checkpoint downloads and achieving efficient checkpoint loading. ServerlessLLM achieves this through three main contributions: (i) fast LLM checkpoint loading via a novel loading-optimized checkpoint format design, coupled with an efficient multi-tier checkpoint loading system; (ii) locality-driven LLM inference with live migration, which allows ServerlessLLM to effectively achieve locality-driven server allocation while preserving the low latency of ongoing LLM inference; and (iii) locality-aware server allocation, enabling ServerlessLLM to evaluate the status of each server in a cluster and effectively schedule model startup time to capitalize on local checkpoint placement. Our comprehensive experiments, which include microbenchmarks and real-world traces, show that ServerlessLLM surpasses state-of-the-art systems by 10 - 200X in latency performance when running various LLM inference workloads.

翻译：本文提出了ServerlessLLM，一种面向大语言模型的位置增强无服务器推理系统。该系统利用GPU服务器上存储与内存设备的大容量及高带宽特性，有效减少昂贵的远程检查点下载开销，实现高效的检查点加载。ServerlessLLM通过三项核心贡献实现上述目标：（一）创新的加载优化检查点格式设计，结合高效的多层级检查点加载系统，实现快速的LLM检查点加载；（二）基于位置驱动的LLM推理与实时迁移技术，使系统能够有效实现位置驱动的服务器分配，同时保持正在运行的LLM推理的低延迟特性；（三）位置感知的服务器分配机制，通过评估集群中各服务器状态并合理规划模型启动时间，充分利用本地检查点部署优势。涵盖微基准测试与真实轨迹的综合实验表明，在不同LLM推理工作负载下，ServerlessLLM在延迟性能上超越现有最先进系统10-200倍。