As system parallelism at chip- and server-level increases, challenges that arose with network-level systems a decade ago, are now being encountered with these massively parallel systems that have become an important workhorse for Machine Learning workloads as well as Graph and Sparse workloads. To tackle the communication bottlenecks, recent works have introduced task-based parallelization schemes to accelerate graph search and sparse data-structure traversal, where some solutions scale up to thousands of processing units (PUs) on a single chip. However, existing communication schemes do not scale to larger than thousands of processing tiles. To address these challenges we propose Tascade, a system that offers hardware-supported, efficient and balanced reduction trees to reduce communication overheads in task-based parallelization schemes and scales up to a million PUs. Tascade achieves this by implementing an execution model utilizing proxy regions and cascading updates, along with a supporting hardware design that enables the execution of the reduction tree at the chip level. The Tascade approach reduces overall communication and improves load balancing. We evaluate six applications and four datasets to provide a detailed analysis of Tascade's performance, power, and traffic-reduction gains over prior work. Our parallelization of Breadth-First-Search with RMAT-26 across a million PUs, the largest of the literature, reaches 5305 GTEPS.

翻译：随着芯片级与服务器级系统并行度的提升，十年前网络级系统面临的挑战如今在已成为机器学习、图计算及稀疏计算重要平台的大规模并行系统中再次出现。为缓解通信瓶颈，近期研究引入基于任务的并行化方案加速图搜索与稀疏数据结构遍历，部分方案已在单芯片上扩展至数千个处理单元。然而现有通信架构无法支持超数千处理瓦片的规模扩展。针对上述问题，本文提出Tascade系统，通过硬件支持的高效均衡归约树降低任务并行化方案中的通信开销，可扩展至百万级处理单元。Tascade采用代理区域和级联更新机制的执行模型，配合支持芯片级归约树执行的硬件设计方案，有效降低整体通信开销并改善负载均衡。我们基于六个应用和四个数据集的评估，详细分析了Tascade相比现有工作的性能、功耗及流量缩减优势。其中，采用RMAT-26数据集的广度优先搜索在百万处理单元上的并行实现达到5305 GTEPS，创下文献最大规模记录。