With deep neural networks (DNNs) emerging as the backbone in a multitude of computer vision tasks, their adoption in real-world consumer applications broadens continuously. Given the abundance and omnipresence of smart devices, "smart ecosystems" are being formed where sensing happens simultaneously rather than standalone. This is shifting the on-device inference paradigm towards deploying centralised neural processing units (NPUs) at the edge, where multiple devices (e.g. in smart homes or autonomous vehicles) can stream their data for processing with dynamic rates. While this provides enhanced potential for input batching, naive solutions can lead to subpar performance and quality of experience, especially under spiking loads. At the same time, the deployment of dynamic DNNs, comprising stochastic computation graphs (e.g. early-exit (EE) models), introduces a new dimension of dynamic behaviour in such systems. In this work, we propose a novel early-exit-aware scheduling algorithm that allows sample preemption at run time, to account for the dynamicity introduced both by the arrival and early-exiting processes. At the same time, we introduce two novel dimensions to the design space of the NPU hardware architecture, namely Fluid Batching and Stackable Processing Elements, that enable run-time adaptability to different batch sizes and significantly improve the NPU utilisation even at small batch sizes. Our evaluation shows that our system achieves an average 1.97x and 6.7x improvement over state-of-the-art DNN streaming systems in terms of average latency and tail latency SLO satisfaction, respectively.

翻译：随着深心神经网络(DNNS)逐渐成为大量计算机视觉任务的主干,它们被应用于现实世界消费者应用中的可能性不断扩大。鉴于智能设备的丰富性和无孔不入,“智能生态系统”正在形成,同时进行感测而不是独立运行。这正在将“智能生态系统”的“智能生态系统”模式改变为在边缘部署中央神经处理器(NPUs)的新层面,在边缘部署中央神经处理器(NPUs),多设备(例如智能家庭或自主车辆)可以以动态速率输送其数据以供处理。虽然这为投入分批提供了更大的潜力,但天真的解决方案可以导致低效和经验质量,特别是在蒸汽负荷负荷下。与此同时,动态DNNNNNNNNNNS值的部署,同时包括随机计算图(例如早期输出(EEE)模型)模型)的部署,在这种系统中引入了动态行为的新层面。在这项工作中,我们提出了一种新型的早期感知度列表算法,让样本在运行的流中, 将N-deexinal-deal-deal-deal Stal Stal-deal-dealtial-deal-staltial-de Stal-stal-deal-deal-destrutal-destrututilutal strutal strutal strutal-stal siax strutal-stal siutal siutal sial sial siutal sium sial sial si) 使我们的系统在正常结构结构结构化结构结构结构结构结构结构结构结构结构结构化系统大大推进两个新进入和智能化系统升级。