Video and image streaming on edge devices requires low latency. To address this, Neural Networks (NNs) are widely used, and prior work mainly focuses on accelerating them with single hardware units such as Graphics Processing Units (GPUs), Field Programmable Gate Arrays (FPGAs), and Deep Learning Processing Units (DPUs). However, further reductions in latency can be observed by combining these units. In this paper, partitioning CNN inference across DPU and GPU (Split CNN Inference) is proposed. The first partition runs on the AI engines (DPU) of a Versal VCK190, which consists of initial CNN layers processing the input images. The DPU processes the first partition near the source of the data. Pipelined asynchronously, a GPU runs the remaining layers. The GPU (NVIDIA RTX 2080) processes the second partition, albeit having reduced the data transfer between the data source (storage/camera) and the GPU. Furthermore, a Graph Neural Network (GNN)-based partition index prediction method is proposed to automate the partitioning of CNNs needed for the Split Inference. Well established models such as LeNet-5, ResNet18/50/101/152, VGG16, and MobileNetv2 are analyzed. Results demonstrate up to 2.48x latency improvement over DPU-only execution and up to 3.37x over GPU-only execution. The trained GNN model splits the layers between the appropriate devices with 96.27% accuracy.

翻译：边缘设备上的视频和图像流处理要求低延迟。为此，神经网络（NN）被广泛使用，现有研究主要集中于利用单一硬件单元（如图形处理器GPU、现场可编程门阵列FPGA、深度学习处理器DPU）加速推理。然而，通过组合这些单元可进一步降低延迟。本文提出在DPU和GPU上分区执行CNN推理（分裂式CNN推理）。第一个分区运行于Versal VCK190的AI引擎（DPU）上，负责处理输入图像的初始CNN层；DPU在数据源附近处理第一分区。通过异步流水线，GPU（NVIDIA RTX 2080）运行剩余层，同时减少了数据源（存储/摄像头）与GPU间的数据传输量。此外，提出基于图神经网络（GNN）的分区索引预测方法，以自动实现分裂式推理所需的CNN分区。分析涵盖LeNet-5、ResNet18/50/101/152、VGG16和MobileNetv2等经典模型。结果表明，相较于纯DPU执行，延迟最高提升2.48倍；相较于纯GPU执行，延迟最高提升3.37倍。训练后的GNN模型以96.27%的准确率在设备间划分网络层。