Hardware accelerators such as GPUs are required for real-time, low-latency inference with Deep Neural Networks (DNN). However, due to the inherent limits to the parallelism they can exploit, DNNs often under-utilize the capacity of today's high-end accelerators. Although spatial multiplexing of the GPU, leads to higher GPU utilization and higher inference throughput, there remain a number of challenges. Finding the GPU percentage for right-sizing the GPU for each DNN through profiling, determining an optimal batching of requests to balance throughput improvement while meeting application-specific deadlines and service level objectives (SLOs), and maximizing throughput by appropriately scheduling DNNs are still significant challenges. This paper introduces a dynamic and fair spatio-temporal scheduler (D-STACK) that enables multiple DNNs to run in the GPU concurrently. To help allocate the appropriate GPU percentage (we call it the "Knee"), we develop and validate a model that estimates the parallelism each DNN can utilize. We also develop a lightweight optimization formulation to find an efficient batch size for each DNN operating with D-STACK. We bring together our optimizations and our spatio-temporal scheduler to provide a holistic inference framework. We demonstrate its ability to provide high throughput while meeting application SLOs. We compare D-STACK with an ideal scheduler that can allocate the right GPU percentage for every DNN kernel. D-STACK gets higher than 90 percent throughput and GPU utilization compared to the ideal scheduler. We also compare D-STACK with other GPU multiplexing and scheduling methods (e.g., NVIDIA Triton, Clipper, Nexus), using popular DNN models. Our controlled experiments with multiplexing several popular DNN models achieve up to 1.6X improvement in GPU utilization and up to 4X improvement in inference throughput.

翻译：硬件加速器（如GPU）对于深度神经网络（DNN）的实时低延迟推理不可或缺。然而，由于DNN可开发的内在并行性存在固有局限，其往往难以充分利用当今高端加速器的计算能力。尽管GPU的空间多路复用能提高GPU利用率和推理吞吐量，但仍面临一系列挑战：通过性能剖析确定适配每个DNN的GPU分配比例（即"右规模"）、确定最优请求批处理大小以平衡吞吐量提升与满足应用特定的截止时间和服务水平目标（SLO），以及通过合理调度DNN最大化吞吐量，仍是亟待解决的关键问题。本文提出了一种动态公平的时空调度器（D-STACK），支持多个DNN在GPU上并发运行。为辅助分配恰当的GPU比例（称为"Knee"），我们开发并验证了一个估算每个DNN可开发并行性的模型。同时，我们设计了一种轻量级优化公式来寻找D-STACK运行环境下每个DNN的高效批处理大小。通过整合优化方法与时空调度器，我们构建了一个整体推理框架，验证了其在满足应用SLO的同时实现高吞吐量的能力。与可为每个DNN内核分配精确GPU比例的理想调度器相比，D-STACK的吞吐量和GPU利用率均达到90%以上。我们还将D-STACK与主流GPU多路复用及调度方法（如NVIDIA Triton、Clipper、Nexus）进行对比，使用经典DNN模型的受控实验表明：GPU利用率提升达1.6倍，推理吞吐量提升达4倍。