GPGPU-based clusters and supercomputers have become extremely popular in the last ten years. There is a large number of GPGPU hardware counters exposed to the users, however, very little analysis has been done regarding insights they might offer about workloads running on them. In this work, we address this gap by analyzing previously unexplored GPU hardware counters collected via Lightweight Distributed Metric Service on Perlmutter, a leadership-class supercomputer. We examine several hardware counters related to utilization of GPU cores and memory and present a detailed spatial and temporal analysis of GPU workloads. We investigate spatial imbalance -- uneven GPU usage across multiple GPUs within a job. Our temporal study examines how GPU usage fluctuates during a job's lifetime, introducing two new metrics -- burstiness (the irregularity of large utilization changes) and temporal imbalance (deviations from mean utilization over time). Additionally, we compare machine learning and traditional high performance computing jobs. Our findings uncover inefficiencies and imbalances that can inform workload optimization and future HPC system design.

翻译：基于GPGPU的集群和超级计算机在过去十年中变得极为流行。虽然向用户暴露了大量的GPGPU硬件性能计数器，但关于这些计数器可能为运行其上的工作负载提供何种洞察，目前仍鲜有分析。本研究通过分析在领导级超级计算机Perlmutter上通过轻量级分布式度量服务收集的、此前未被探索的GPU硬件计数器，填补了这一空白。我们考察了与GPU核心及内存利用率相关的多个硬件计数器，并对GPU工作负载进行了细致的时空分析。我们研究了空间不平衡性——即单个作业内多个GPU之间的使用不均衡现象。我们的时间分析探究了GPU使用率在作业生命周期内的波动情况，并引入了两个新指标：突发性（大幅利用率变化的不规则程度）和时间不平衡性（随时间推移相对于平均利用率的偏离程度）。此外，我们还比较了机器学习与传统高性能计算作业。我们的研究结果揭示了可能影响工作负载优化及未来HPC系统设计的低效与不平衡问题。