Serverless computing has emerged as a compelling solution for cloud-based model inference. However, as modern large language models (LLMs) continue to grow in size, existing serverless platforms often face substantial model startup overhead. This poses a significant challenge in efficiently scaling model instances to accommodate dynamic, bursty workloads commonly observed in real-world inference services. In this paper, we introduce λScale, an efficient serverless inference system to achieve fast model scaling. The key idea behind λScale is to leverage high-speed RDMA networks between GPU nodes for fast model multicast, while enabling distributed inference execution during model transmission -- referred to as "execute-while-load". λScale proposes an efficient model scaling scheme, λPipe, which supports adaptive model multicast and dynamically constructs execution pipelines across receiving nodes for collaborative, distributed inference. Additionally, λScale supports efficient model management across GPU and host memory, allowing fast scaling for models across different storage tiers. Evaluation results show that λScale enables fast model scaling and effectively handles load spikes, achieving up to 5x tail-latency improvement and 31.3% cost reduction compared to state-of-the-art solutions on real-world LLM inference traces.

翻译：无服务器计算已成为基于云的模型推理的一种引人注目的解决方案。然而，随着现代大语言模型（LLMs）规模持续增长，现有的无服务器平台常常面临巨大的模型启动开销。这为高效扩缩模型实例以应对实际推理服务中常见的动态、突发性工作负载带来了重大挑战。本文介绍了λScale，一个高效的无服务器推理系统，旨在实现快速的模型扩缩容。λScale的核心思想是利用GPU节点间的高速RDMA网络进行快速模型组播，同时在模型传输过程中启用分布式推理执行——我们称之为“边加载边执行”。λScale提出了一种高效的模型扩缩方案λPipe，它支持自适应模型组播，并在接收节点间动态构建执行流水线，以实现协作式分布式推理。此外，λScale支持跨GPU和主机内存的高效模型管理，允许对不同存储层级的模型进行快速扩缩容。评估结果表明，λScale能够实现快速的模型扩缩容，并有效处理负载峰值。在实际LLM推理负载跟踪上，与最先进的解决方案相比，λScale实现了高达5倍的尾部延迟改进和31.3%的成本降低。