It is commonly assumed that the end-to-end networking performance of edge offloading is purely dictated by that of the network connectivity between end devices and edge computing facilities, where ongoing innovation in 5G/6G networking can help. However, with the growing complexity of edge-offloaded computation and dynamic load balancing requirements, an offloaded task often goes through a multi-stage pipeline that spans across multiple compute nodes and proxies interconnected via a dedicated network fabric within a given edge computing facility. As the latest hardware-accelerated transport technologies such as RDMA and GPUDirect RDMA are adopted to build such network fabric, there is a need for good understanding of the full potential of these technologies in the context of computation offload and the effect of different factors such as GPU scheduling and characteristics of computation on the net performance gain achievable by these technologies. This paper unveils detailed insights into the latency overhead in typical machine learning (ML)-based computation pipelines and analyzes the potential benefits of adopting hardware-accelerated communication. To this end, we build a model-serving framework that supports various communication mechanisms. Using the framework, we identify performance bottlenecks in state-of-the-art model-serving pipelines and show how hardware-accelerated communication can alleviate them. For example, we show that GPUDirect RDMA can save 15--50\% of model-serving latency, which amounts to 70--160 ms.

翻译：通常认为，边缘卸载的端到端网络性能完全由终端设备与边缘计算设施之间的网络连接决定，而5G/6G网络的持续创新可对此提供助力。然而，随着边缘卸载计算日益复杂以及动态负载均衡需求的增长，卸载任务通常需经过多阶段流水线，该流水线跨越同一边缘计算设施内通过专用网络结构互联的多个计算节点和代理。随着RDMA和GPUDirect RDMA等最新硬件加速传输技术被用于构建此类网络结构，亟需深入理解这些技术在计算卸载场景中的全部潜力，以及GPU调度、计算特性等不同因素对上述技术可实现净性能增益的影响。本文揭示了典型基于机器学习（ML）计算流水线的延迟开销细节，并分析了采用硬件加速通信的潜在优势。为此，我们构建了一个支持多种通信机制的模型服务框架。利用该框架，我们识别了当前最先进模型服务流水线的性能瓶颈，并展示了硬件加速通信如何缓解这些问题。例如，我们证明GPUDirect RDMA可节省15%-50%的模型服务延迟，相当于70-160毫秒。