Large-scale GPU clusters are widely-used to speed up both latency-critical (online) and best-effort (offline) deep learning (DL) workloads. However, most DL clusters either dedicate each GPU to one workload or share workloads in time, leading to very low GPU resource utilization. We present MuxFlow, the first production cluster system that supports efficient and safe space-sharing for DL workloads. NVIDIA MPS provides an opportunity to share multiple workloads in space on widely-deployed NVIDIA GPUs, but it cannot guarantee the performance and safety of online workloads. MuxFlow introduces a two-level protection mechanism for memory and computation to guarantee the performance of online workloads. Based on our practical error analysis, we design a mixed error-handling mechanism to guarantee the safety of online workloads. MuxFlow further proposes dynamic streaming multiprocessor (SM) allocation and matching-based scheduling to improve the efficiency of offline workloads. MuxFlow has been deployed at CompanyX's clusters with more than 20,000 GPUs. The deployment results indicate that MuxFlow substantially improves the GPU utilization from 26$\%$ to 76$\%$, SM activity from 16$\%$ to 33$\%$, and GPU memory from 42$\%$ to 48$\%$.

翻译：大规模GPU集群广泛用于加速对延迟敏感（在线）和尽力而为型（离线）深度学习工作负载。然而，大多数深度学习集群要么将每个GPU专用于单一工作负载，要么按时间分片共享工作负载，导致GPU资源利用率极低。我们提出MuxFlow，这是首个支持深度学习工作负载高效安全空间共享的生产集群系统。NVIDIA MPS提供了在广泛部署的NVIDIA GPU上实现多工作负载空间共享的契机，但无法保证在线工作负载的性能与安全性。MuxFlow引入内存与计算双层保护机制，保障在线工作负载的性能。基于实际错误分析，我们设计了混合错误处理机制来确保在线工作负载的安全性。MuxFlow进一步提出动态流式多处理器（SM）分配与基于匹配的调度机制，以提升离线工作负载的效率。MuxFlow已在拥有超过20,000个GPU的CompanyX集群中部署。部署结果表明，MuxFlow将GPU利用率从26%提升至76%，SM活动率从16%提升至33%，GPU内存占用率从42%提升至48%。