It is commonly assumed that the end-to-end networking performance of edge offloading is purely dictated by that of the network connectivity between end devices and edge computing facilities, where ongoing innovation in 5G/6G networking can help. However, with the growing complexity of edge-offloaded computation and dynamic load balancing requirements, an offloaded task often goes through a multi-stage pipeline that spans across multiple compute nodes and proxies interconnected via a dedicated network fabric within a given edge computing facility. As the latest hardware-accelerated transport technologies such as RDMA and GPUDirect RDMA are adopted to build such network fabric, there is a need for good understanding of the full potential of these technologies in the context of computation offload and the effect of different factors such as GPU scheduling and characteristics of computation on the net performance gain achievable by these technologies. This paper unveils detailed insights into the latency overhead in typical machine learning (ML)-based computation pipelines and analyzes the potential benefits of adopting hardware-accelerated communication. To this end, we build a model-serving framework that supports various communication mechanisms. Using the framework, we identify performance bottlenecks in state-of-the-art model-serving pipelines and show how hardware-accelerated communication can alleviate them. For example, we show that GPUDirect RDMA can save 15--50\% of model-serving latency, which amounts to 70--160 ms.

翻译：通常认为，边缘卸载的端到端网络性能完全由终端设备与边缘计算设施之间的网络连接决定，而5G/6G网络的持续创新有助于提升这一性能。然而，随着边缘卸载计算复杂度的增加和动态负载均衡需求的提高，卸载任务往往需要经过多阶段流水线，该流水线跨越多个计算节点和代理，这些节点和代理通过边缘计算设施内部的专用网络结构互连。随着RDMA和GPUDirect RDMA等最新硬件加速传输技术被用于构建此类网络结构，有必要深入理解这些技术在计算卸载背景下的全部潜力，以及GPU调度、计算特性等不同因素对这些技术所能实现的净性能增益的影响。本文揭示了典型机器学习计算流水线中的延迟开销细节，并分析了采用硬件加速通信的潜在优势。为此，我们构建了一个支持多种通信机制的模型服务框架。利用该框架，我们识别了最先进的模型服务流水线中的性能瓶颈，并展示了硬件加速通信如何缓解这些瓶颈。例如，我们证明GPUDirect RDMA可节省15%–50%的模型服务延迟，相当于70–160毫秒。