LLM inference powers latency-critical production services nowadays. The bursty nature of inference traffic results in over-provisioning, which in turn leads to resource underutilization. While online-offline colocation promises to utilize idle capacity, broad production deployment must overcome two major challenges: (i) large online interference due to slow or frequent preemptions, and (ii) extensive frameworks and drivers modifications, to colocate different models and support preemptions. We present Valve, a production-friendly colocation system that jointly bounds preemption latency and preemption rate. Specifically, Valve enables sub-millisecond compute preemption at most once per online request, and rate-limited sub-layer memory reclamation. These guaranties are provided by a GPU runtime that combines channel-controlled compute isolation, page-fault-free memory reclamation, and dynamic memory reservation. Critically, Valve is practical to deploy, requiring one line of driver modification and 20 lines of framework patch. Deployed on 8,054 GPUs in production, Valve improves cluster utilization by 34.6%, which translates to a 2,170 GPU save. This efficiency gains is achieved with minimal online interference, incurring <5% TTFT increase and <2% TPOT increase across workloads.

翻译：LLM推理如今支撑着延迟敏感的生产级服务。推理流量的突发性导致过度配置，进而造成资源利用率低下。虽然在线-离线共置有望利用空闲算力，但大规模部署必须克服两大挑战：(i) 因缓慢或频繁的抢占导致严重的在线干扰；(ii) 需要修改大量框架和驱动程序以支持不同模型的共置与抢占。我们提出Valve，一种生产级共置系统，可联合限制抢占延迟与抢占速率。具体而言，Valve实现每个在线请求最多一次亚毫秒级计算抢占，以及速率受限的子层内存回收。这些保障通过结合通道控制的计算隔离、无缺页内存回收与动态内存预留的GPU运行时实现。关键的是，Valve具备实用部署性，仅需修改一行驱动程序与20行框架补丁。在8054块生产级GPU上部署后，Valve将集群利用率提升34.6%，相当于节省2170块GPU。这一效率增益带来的在线干扰极小，各工作负载的TTFT增长低于5%，TPOT增长低于2%。