Emerging real-time multi-model ML (RTMM) workloads such as AR/VR and drone control involve dynamic behaviors in various granularity; task, model, and layers within a model. Such dynamic behaviors introduce new challenges to the system software in an ML system since the overall system load is not completely predictable, unlike traditional ML workloads. In addition, RTMM workloads require real-time processing, involve highly heterogeneous models, and target resource-constrained devices. Under such circumstances, developing an effective scheduler gains more importance to better utilize underlying hardware considering the unique characteristics of RTMM workloads. Therefore, we propose a new scheduler, DREAM, which effectively handles various dynamicity in RTMM workloads targeting multi-accelerator systems. DREAM quantifies the unique requirements for RTMM workloads and utilizes the quantified scores to drive scheduling decisions, considering the current system load and other inference jobs on different models and input frames. DREAM utilizes tunable parameters that provide fast and effective adaptivity to dynamic workload changes. In our evaluation of five scenarios of RTMM workload, DREAM reduces the overall UXCost, which is an equivalent metric of the energy-delay product (EDP) for RTMM defined in the paper, by 32.2% and 50.0% in the geometric mean (up to 80.8% and 97.6%) compared to state-of-the-art baselines, which shows the efficacy of our scheduling methodology.

翻译：新兴的实时多模型机器学习（RTMM）工作负载（如增强现实/虚拟现实和无人机控制）在多个粒度层面（包括任务、模型以及模型内部层级）展现出动态行为。与传统机器学习工作负载不同，此类动态行为给机器学习系统的系统软件带来了新挑战，因为整体系统负载无法完全预测。此外，RTMM工作负载需要实时处理、涉及高度异构的模型，并面向资源受限设备。在此背景下，鉴于RTMM工作负载的独特特性，开发高效调度器以更优地利用底层硬件变得尤为重要。为此，我们提出新型调度器DREAM，它能有效应对面向多加速器系统的RTMM工作负载中的多种动态性。DREAM量化了RTMM工作负载的独特需求，并利用量化得分驱动调度决策，同时考虑当前系统负载及其他推理作业（针对不同模型和输入帧）。DREAM采用可调参数，能快速有效适应动态工作负载变化。在五种RTMM工作负载场景的评估中，与当前最先进的基线方法相比，DREAM将整体用户体验成本（UXCost，本文定义的RTMM工作负载中能量延迟积的等效指标）的几何平均值分别降低32.2%和50.0%（最高达80.8%和97.6%），验证了所提调度方法的有效性。