Graphics processing units (GPUs) are widely used in many high-performance computing (HPC) applications such as imaging/video processing and training deep-learning models in artificial intelligence. GPUs installed in HPC systems are often heavily used, and GPU failures occur during HPC system operations. Thus, the reliability of GPUs is of interest for the overall reliability of HPC systems. The Cray XK7 Titan supercomputer was one of the top ten supercomputers in the world. The failure event times of more than 30,000 GPUs in Titan were recorded and previous data analysis suggested that the failure time of a GPU may be affected by the GPU's connectivity location inside the supercomputer among other factors. In this paper, we conduct in-depth statistical modeling of GPU failure times to study the effect of location on GPU failures under competing risks with covariates and spatially correlated random effects. In particular, two major failure types of GPUs in Titan are considered. The connectivity locations of cabinets are modeled as spatially correlated random effects, and the positions of GPUs inside each cabinet are treated as covariates. A Bayesian framework is used for statistical inference. We also compare different methods of estimation such as the maximum likelihood, which is implemented via an expectation-maximization algorithm. Our results provide interesting insights into GPU failures in HPC systems.

翻译：图形处理单元（GPU）广泛用于许多高性能计算（HPC）应用中，如图像/视频处理及人工智能深度学习模型的训练。HPC系统中安装的GPU常被高强度使用，并且在HPC系统运行过程中会发生GPU故障。因此，GPU的可靠性对HPC系统的整体可靠性至关重要。Cray XK7 Titan超级计算机曾是全球十大超级计算机之一。该系统中超过30,000个GPU的故障事件时间被记录，先前数据分析表明，GPU的故障时间可能受其在超级计算机内部的连接位置等因素影响。本文对GPU故障时间进行深入统计建模，研究在存在协变量和空间相关随机效应的竞争风险下，位置对GPU故障的影响。特别地，我们考虑了Titan中GPU的两种主要故障类型。机柜的连接位置被建模为空间相关随机效应，而每个机柜内GPU的位置则作为协变量处理。采用贝叶斯框架进行统计推断。我们还比较了不同估计方法，例如通过期望最大化算法实现的极大似然估计。我们的结果为HPC系统中的GPU故障提供了有价值的见解。