Current supercomputers often have a heterogeneous architecture using both CPUs and GPUs. At the same time, numerical simulation tasks frequently involve multiphysics scenarios whose components run on different hardware due to multiple reasons, e.g., architectural requirements, pragmatism, etc. This leads naturally to a software design where different simulation modules are mapped to different subsystems of the heterogeneous architecture. We present a detailed performance analysis for such a hybrid four-way coupled simulation of a fully resolved particle-laden flow. The Eulerian representation of the flow utilizes GPUs, while the Lagrangian model for the particles runs on CPUs. First, a roofline model is employed to predict the node level performance and to show that the lattice-Boltzmann-based fluid simulation reaches very good performance on a single GPU. Furthermore, the GPU-GPU communication for a large-scale flow simulation results in only moderate slowdowns due to the efficiency of the CUDA-aware MPI communication, combined with communication hiding techniques. On 1024 A100 GPUs, a parallel efficiency of up to 71% is achieved. While the flow simulation has good performance characteristics, the integration of the stiff Lagrangian particle system requires frequent CPU-CPU communications that can become a bottleneck. Additionally, special attention is paid to the CPU-GPU communication overhead since this is essential for coupling the particles to the flow simulation. However, thanks to our problem-aware co-partitioning, the CPU-GPU communication overhead is found to be negligible. As a lesson learned from this development, four criteria are postulated that a hybrid implementation must meet for the efficient use of heterogeneous supercomputers. Additionally, an a priori estimate of the speedup for hybrid implementations is suggested.

翻译：当前超级计算机常采用同时包含CPU与GPU的异构架构。与此同时，数值模拟任务常涉及多物理场场景，其各组成部分因架构需求、实用性等多重原因需在不同硬件上运行。这自然催生了一种软件设计范式：将不同模拟模块映射至异构架构的不同子系统。本文针对全解析含颗粒流动的混合四向耦合模拟进行了详尽的性能分析。其中流动的欧拉描述在GPU上执行，而颗粒的拉格朗日模型在CPU上运行。首先，采用屋顶模型预测节点级性能，结果表明基于格子玻尔兹曼方法的流体模拟在单GPU上能达到优异性能。此外，大规模流动模拟的GPU间通信仅造成适度性能下降，这得益于CUDA-aware MPI通信的高效性与通信隐藏技术的结合。在1024个A100 GPU上实现了高达71%的并行效率。尽管流动模拟具备良好的性能特征，刚性拉格朗日颗粒系统的集成需要频繁的CPU间通信，这可能成为性能瓶颈。同时，本文特别关注CPU-GPU通信开销，因其对颗粒与流动模拟的耦合至关重要。然而，通过采用面向问题的协同分区策略，CPU-GPU通信开销可忽略不计。基于此开发经验，我们提出异构超级计算机高效利用需满足的四项混合实现准则，并建议了混合实现加速比的先验估算方法。