In this paper, we conduct systematic measurement studies to show that the high memory bandwidth consumption of modern distributed applications can lead to a significant drop of network throughput and a large increase of tail latency in high-speed RDMA networks.We identify its root cause as the high contention of memory bandwidth between application processes and network processes. This contention leads to frequent packet drops at the NIC of receiving hosts, which triggers the congestion control mechanism of the network and eventually results in network performance degradation. To tackle this problem, we make a key observation that given the distributed storage service, the vast majority of data it receives from the network will be eventually written to high-speed storage media (e.g., SSD) by CPU. As such, we propose to bypass host memory when processing received data to completely circumvent this performance bottleneck. In particular, we design Lamda, a novel receiver cache processing system that consumes a small amount of CPU cache to process received data from the network at line rate. We implement a prototype of Lamda and evaluate its performance extensively in a Clos-based testbed. Results show that for distributed storage applications, Lamda improves network throughput by 4.7% with zero memory bandwidth consumption on storage nodes, and improves network throughput by up 17% and 45% for large block size and small size under the memory bandwidth pressure, respectively. Lamda can also be applied to latency-sensitive HPC applications, which reduces their communication latency by 35.1%.

翻译：本文通过系统性测量研究，揭示现代分布式应用的高内存带宽消耗会导致高速RDMA网络中网络吞吐量显著下降和尾部延迟大幅增加。我们确定其根本原因是应用程序进程与网络进程之间的内存带宽高度竞争。这种竞争导致接收主机网卡频繁丢包，触发网络拥塞控制机制，最终造成网络性能退化。为解决该问题，我们提出关键洞察：在分布式存储服务中，网络接收的绝大多数数据最终将由CPU写入高速存储介质（如SSD）。因此，我们建议在处理接收数据时绕过主机内存，以彻底规避此性能瓶颈。具体而言，我们设计了Lamda——一种新型接收端缓存处理系统，该系统仅消耗少量CPU缓存即可在线速下处理网络接收数据。我们在基于Clos结构的测试平台上实现了Lamda原型并进行了广泛性能评估。结果表明：对于分布式存储应用，Lamda在存储节点零内存带宽消耗条件下将网络吞吐量提升4.7%；在内存带宽压力下，大块数据和小块数据的吞吐量分别提升17%和45%。Lamda还可应用于延迟敏感的HPC应用，将其通信延迟降低35.1%。