This paper presents \pandora, a novel parallel algorithm for efficiently constructing dendrograms for single-linkage hierarchical clustering, including \hdbscan. Traditional dendrogram construction methods from a minimum spanning tree (MST), such as agglomerative or divisive techniques, often fail to efficiently parallelize, especially with skewed dendrograms common in real-world data. \pandora addresses these challenges through a unique recursive tree contraction method, which simplifies the tree for initial dendrogram construction and then progressively reconstructs the complete dendrogram. This process makes \pandora asymptotically work-optimal, independent of dendrogram skewness. All steps in \pandora are fully parallel and suitable for massively threaded accelerators such as GPUs. Our implementation is written in Kokkos, providing support for both CPUs and multi-vendor GPUs (e.g., Nvidia, AMD). The multithreaded version of \pandora is 2.2$\times$ faster than the current best-multithreaded implementation, while the GPU \pandora implementation achieved 6-20$\times$ on \amdgpu and 10-37$\times$ on \nvidiagpu speed-up over multithreaded \pandora. These advancements lead to up to a 6-fold speedup for \hdbscan on GPUs over the current best, which only offload MST construction to GPUs and perform multithreaded dendrogram construction.

翻译：本文提出\pandora——一种用于高效构建单链接层次聚类（包括\hdbscan）树状图的新型并行算法。传统基于最小生成树（MST）的树状图构建方法（如凝聚式或分裂式技术）难以高效并行化，尤其面对真实数据中常见的偏斜树状图时。\pandora通过独特的递归树收缩方法解决这些挑战：先简化树以初步构建树状图，再逐步重构完整树状图。该过程使\pandora在渐进工作最优性上与树状图偏斜度无关。\pandora的所有步骤完全并行，适用于GPU等大规模多线程加速器。我们的实现基于Kokkos框架，同时支持CPU与多厂商GPU（如Nvidia、AMD）。\pandora的多线程版本比当前最优多线程实现快2.2倍；在GPU上，\pandora在\amdgpu和\nvidiagpu上分别实现6-20倍和10-37倍于多线程\pandora的加速。这些进展使\hdbscan在GPU上比当前仅将MST构建卸载至GPU而仍在CPU多线程构建树状图的最优方法快6倍。