Video and image streaming on edge devices requires low latency. To address this, Neural Networks (NNs) are widely used, and prior work mainly focuses on accelerating them with single hardware units such as Graphics Processing Units (GPUs), Field Programmable Gate Arrays (FPGAs), and Deep Learning Processing Units (DPUs). However, further reductions in latency can be observed by combining these units. In this paper, partitioning CNN inference across DPU and GPU (Split CNN Inference) is proposed. The first partition runs on the AI engines (DPU) of a Versal VCK190, which consists of initial CNN layers processing the input images. The DPU processes the first partition near the source of the data. Pipelined asynchronously, a GPU runs the remaining layers. The GPU (NVIDIA RTX 2080) processes the second partition, albeit having reduced the data transfer between the data source (storage/camera) and the GPU. Furthermore, a Graph Neural Network (GNN)-based partition index prediction method is proposed to automate the partitioning of CNNs needed for Split Inference. Well established models such as LeNet-5, ResNet18/50/101/152, VGG16, and MobileNetv2 are analyzed. Results demonstrate up to 2.48x latency improvement over DPU-only execution and up to 3.37x over GPU-only execution. The trained GNN model splits the layers between the appropriate devices with 96.27% accuracy.

翻译：边缘设备上的视频与图像流处理需要低延迟。为此，神经网络（NN）被广泛使用，先前研究主要集中于使用单一硬件单元（如图形处理器GPU、现场可编程门阵列FPGA、深度学习处理器DPU）加速推理。然而，通过组合这些单元可进一步降低延迟。本文提出将CNN推理在DPU与GPU间进行分裂（分裂式CNN推理）。第一个分区在Versal VCK190的AI引擎（DPU）上运行，包含处理输入图像的初始CNN层。DPU在数据源附近处理第一个分区。通过异步流水线方式，GPU运行剩余层。GPU（NVIDIA RTX 2080）处理第二个分区，同时减少了数据源（存储/摄像头）与GPU之间的数据传输量。此外，提出基于图神经网络（GNN）的分区索引预测方法，以实现分裂推理所需的CNN自动分区。分析了LeNet-5、ResNet18/50/101/152、VGG16和MobileNetv2等成熟模型。结果表明，相较于纯DPU执行，延迟最高可提升2.48倍；相较于纯GPU执行，延迟最高可提升3.37倍。训练后的GNN模型以96.27%的准确率将层分配到相应设备。