Parallel software codes in high performance computing (HPC) continue to grow in complexity and scale as we enter the exascale era. A diverse set of emerging hardware and programming paradigms make developing, optimizing, and maintaining parallel software burdensome for developers. One way to alleviate some of these burdens is with automated development and analysis tools. Such tools can perform complex and/or remedial tasks for developers that increase their productivity and decrease the chance for error. So far, such tools for code development and performance analysis have been limited in the complexity of tasks they can perform. However, with recent advancements in language modeling, and the wealth of code related data that is now available online, these tools have started to utilize predictive language models to automate more complex tasks. In this paper, we show how large language models (LLMs) can be applied to tasks specific to high performance and scientific codes. We train LLMs using code and performance data that is specific to parallel codes. We compare several recent LLMs on HPC related tasks and introduce a new model, HPC-Coder, trained on parallel code. In our experiments we show that this model can auto-complete HPC functions where general models cannot, decorate for loops with OpenMP pragmas, and model performance changes in two scientific application repositories.

翻译：随着我们进入百亿亿次计算时代，高性能计算（HPC）中的并行软件代码在复杂性和规模上持续增长。多样化的新兴硬件和编程范式使得并行软件的开发、优化和维护对开发者来说负担沉重。减轻这些负担的一种方法是使用自动化开发和分析工具。这类工具可以为开发者执行复杂和/或补救性任务，从而提高其生产力并降低出错概率。迄今为止，用于代码开发和性能分析的工具在其可执行任务的复杂性方面仍受限制。然而，随着语言建模的最新进展以及如今在线可用的海量代码相关数据，这些工具已开始利用预测性语言模型来自动化更复杂的任务。在本文中，我们展示了如何将大型语言模型（LLMs）应用于高性能和科学代码的特定任务。我们使用并行代码特有的代码和性能数据来训练LLMs。我们对比了多个近期LLMs在HPC相关任务上的表现，并引入了一个基于并行代码训练的新模型——HPC-Coder。实验表明，该模型能够自动补全通用模型无法完成的HPC函数、为循环添加OpenMP制导指令，并在两个科学应用代码库中建模性能变化。