In the realm of distributed systems tasked with managing and processing large-scale graph-structured data, optimizing graph partitioning stands as a pivotal challenge. The primary goal is to minimize communication overhead and runtime cost. However, alongside the computational complexity associated with optimal graph partitioning, a critical factor to consider is memory overhead. Real-world graphs often reach colossal sizes, making it impractical and economically unviable to load the entire graph into memory for partitioning. This is also a fundamental premise in distributed graph processing, where accommodating a graph with non-distributed systems is unattainable. Currently, existing streaming partitioning algorithms exhibit a skew-oblivious nature, yielding satisfactory partitioning results exclusively for specific graph types. In this paper, we propose a novel streaming partitioning algorithm, the Skewness-aware Vertex-cut Partitioner S5P, designed to leverage the skewness characteristics of real graphs for achieving high-quality partitioning. S5P offers high partitioning quality by segregating the graph's edge set into two subsets, head and tail sets. Following processing by a skewness-aware clustering algorithm, these two subsets subsequently undergo a Stackelberg graph game. Our extensive evaluations conducted on substantial real-world and synthetic graphs demonstrate that, in all instances, the partitioning quality of S5P surpasses that of existing streaming partitioning algorithms, operating within the same load balance constraints. For example, S5P can bring up to a 51% improvement in partitioning quality compared to the top partitioner among the baselines. Lastly, we showcase that the implementation of S5P results in up to an 81% reduction in communication cost and a 130% increase in runtime efficiency for distributed graph processing tasks on PowerGraph.

翻译：在负责管理和处理大规模图结构数据的分布式系统领域，优化图分割是一项关键挑战。主要目标是最小化通信开销和运行时成本。然而，除了与最优图分割相关的计算复杂性之外，内存开销也是一个关键因素。现实世界的图通常规模庞大，将整个图加载到内存中进行分割既不切实际也不经济。这也是分布式图处理的基本前提，即非分布式系统无法容纳大型图。当前，现有的流式分割算法具有偏斜无关的特性，仅对特定类型的图产生令人满意的分割结果。本文提出了一种新颖的流式分割算法——偏斜感知顶点分割器S5P，旨在利用真实图的偏斜特性实现高质量分割。S5P通过将图的边集分为头集和尾集两个子集来提供高分割质量。在经过偏斜感知聚类算法处理后，这两个子集随后经历一场斯塔克尔伯格图博弈。我们在大量真实世界和合成图上进行的广泛评估表明，在所有情况下，S5P的分割质量均优于现有的流式分割算法，且保持相同的负载平衡约束。例如，与基线中最佳分割器相比，S5P可将分割质量提升高达51%。最后，我们展示了在PowerGraph上实施S5P可使分布式图处理任务的通信成本降低高达81%，运行时效率提升高达130%。