Deep neural networks (DNNs) underpin many machine learning applications. Production quality DNN models achieve high inference accuracy by training millions of DNN parameters which has a significant resource footprint. This presents a challenge for resources operating at the extreme edge of the network, such as mobile and embedded devices that have limited computational and memory resources. To address this, models are pruned to create lightweight, more suitable variants for these devices. Existing pruning methods are unable to provide similar quality models compared to their unpruned counterparts without significant time costs and overheads or are limited to offline use cases. Our work rapidly derives suitable model variants while maintaining the accuracy of the original model. The model variants can be swapped quickly when system and network conditions change to match workload demand. This paper presents DNNShifter, an end-to-end DNN training, spatial pruning, and model switching system that addresses the challenges mentioned above. At the heart of DNNShifter is a novel methodology that prunes sparse models using structured pruning. The pruned model variants generated by DNNShifter are smaller in size and thus faster than dense and sparse model predecessors, making them suitable for inference at the edge while retaining near similar accuracy as of the original dense model. DNNShifter generates a portfolio of model variants that can be swiftly interchanged depending on operational conditions. DNNShifter produces pruned model variants up to 93x faster than conventional training methods. Compared to sparse models, the pruned model variants are up to 5.14x smaller and have a 1.67x inference latency speedup, with no compromise to sparse model accuracy. In addition, DNNShifter has up to 11.9x lower overhead for switching models and up to 3.8x lower memory utilisation than existing approaches.

翻译：深度神经网络（DNN）支撑着众多机器学习应用。工业级DNN模型通过训练数百万参数实现高推理精度，但这也带来了显著的计算资源开销。这给运行在网络边缘的设备（如计算与内存资源受限的移动和嵌入式设备）带来了挑战。为解决这一问题，模型剪枝被用于生成轻量化、更适合这些设备的变体。现有剪枝方法无法在保持与未剪枝模型同等质量的同时避免高昂的时间成本和开销，或仅限于离线场景使用。本研究能够快速生成保持原始模型精度的适合模型变体，并在系统与网络条件变化时快速切换这些变体以适应工作负载需求。本文提出DNNShifter——一个集DNN训练、结构化剪枝与模型切换于一体的端到端系统，用以解决上述挑战。DNNShifter的核心是一种利用结构化剪枝对稀疏模型进行剪枝的新方法论。其生成的剪枝模型变体体积更小、速度更快，在边缘推理场景中优于密集与稀疏模型，同时保持与原始密集模型相近的精度。DNNShifter可生成一整套可根据运行条件快速切换的模型变体组合。与传统训练方法相比，DNNShifter生成剪枝模型变体的速度提升达93倍；与稀疏模型相比，剪枝模型变体体积缩小最高5.14倍，推理延迟加速1.67倍，且不损失稀疏模型精度。此外，DNNShifter的模型切换开销相比现有方法降低最高11.9倍，内存占用降低最高3.8倍。