Parallel programs in high performance computing (HPC) continue to grow in complexity and scale in the exascale era. The diversity in hardware and parallel programming models make developing, optimizing, and maintaining parallel software even more burdensome for developers. One way to alleviate some of these burdens is with automated development and analysis tools. Such tools can perform complex and/or remedial tasks for developers that increase their productivity and decrease the chance for error. Until recently, such tools for code development and performance analysis have been limited in the complexity of tasks they can perform, especially for parallel programs. However, with recent advancements in language modeling, and the availability of large amounts of open-source code related data, these tools have started to utilize predictive language models to automate more complex tasks. In this paper, we show how large language models (LLMs) can be applied to tasks specific to high performance and scientific codes. We introduce a new dataset of HPC and scientific codes and use it to fine-tune several pre-trained models. We compare several pre-trained LLMs on HPC-related tasks and introduce a new model, HPC-Coder, fine-tuned on parallel codes. In our experiments, we show that this model can auto-complete HPC functions where generic models cannot, decorate for loops with OpenMP pragmas, and model performance changes in scientific application repositories as well as programming competition solutions.

翻译：高性能计算（HPC）中的并行程序在后摩尔时代日益复杂且规模持续扩大。硬件与并行编程模型的多样性使得开发、优化与维护并行软件对开发者而言负担更加沉重。缓解这些困难的一种途径是借助自动化开发与分析工具。此类工具可为开发者执行复杂或重复性任务，从而提升其生产力并降低错误概率。直至近期，这类用于代码开发与性能分析的工具仍受限于其可执行任务的复杂性，尤其对于并行程序而言。然而，随着语言建模技术的近期突破以及大规模开源代码相关数据的可用性，这些工具已开始利用预测性语言模型来实现更复杂任务的自动化。本文展示了如何将大型语言模型（LLMs）应用于高性能与科学计算代码的特定任务。我们引入了一个包含HPC与科学代码的新数据集，并利用其对多个预训练模型进行微调。我们比较了多个预训练LLM在HPC相关任务上的表现，并提出了一种专为并行代码微调的新模型——HPC-Coder。实验表明，该模型能补全通用模型无法处理的HPC函数，为循环添加OpenMP指令（pragma），并在科学应用代码库及编程竞赛解决方案中建模性能变化。