Highly parallelized workloads like machine learning training, inferences and general HPC tasks are greatly accelerated using GPU devices. In a cloud computing cluster, serving a GPU's computation power through multi-tasks sharing is highly demanded since there are always more task requests than the number of GPU available. Existing GPU sharing solutions focus on reducing task-level waiting time or task-level switching costs when multiple jobs competing for a single GPU. Non-stopped computation requests come with different priorities, having non-symmetric impact on QoS for sharing a GPU device. Existing work missed the kernel-level optimization opportunity brought by this setting. To address this problem, we present a novel kernel-level scheduling strategy called FIKIT: Filling Inter-kernel Idle Time. FIKIT incorporates task-level priority information, fine-grained kernel identification, and kernel measurement, allowing low priorities task's execution during high priority task's inter-kernel idle time. Thereby, filling the GPU's device runtime fully, and reduce overall GPU sharing impact to cloud services. Across a set of ML models, the FIKIT based inference system accelerated high priority tasks by 1.33 to 14.87 times compared to the JCT in GPU sharing mode, and more than half of the cases are accelerated by more than 3.5 times. Alternatively, under preemptive sharing, the low-priority tasks have a comparable to default GPU sharing mode JCT, with a 0.84 to 1 times ratio. We further limit the kernel measurement and runtime fine-grained kernel scheduling overhead to less than 5%.

翻译：高度并行化的计算任务（如机器学习训练、推理及通用HPC任务）通过GPU设备得以显著加速。在云计算集群中，由于任务请求数量通常超过可用GPU数量，通过多任务共享实现GPU计算能力的高效利用成为迫切需求。现有GPU共享方案主要聚焦于多任务竞争单GPU时的任务级等待时间或任务级切换开销。然而，非中断的计算请求具有不同优先级，对共享GPU设备的QoS产生非对称影响。现有工作未能捕捉到这一场景下内核级优化机会。针对该问题，我们提出一种新型内核级调度策略FIKIT：填充内核间空闲时间。FIKIT融合了任务级优先级信息、细粒度内核识别与内核测量，允许低优先级任务在高优先级任务的内核间空闲时段内执行。由此实现GPU设备运行时完全填充，并降低GPU共享对云服务的整体影响。在多种ML模型上的实验表明，基于FIKIT的推理系统在GPU共享模式下，高优先级任务的加速比相比JCT达到1.33至14.87倍，其中超过半数场景加速比超过3.5倍。而在抢占式共享模式下，低优先级任务的JCT与默认GPU共享模式相当，比值范围为0.84至1倍。此外，我们将内核测量与运行时细粒度内核调度开销控制在5%以内。