The imperative need to scale computation across numerous nodes highlights the significance of efficient parallel computing, particularly in the realm of Message Passing Interface (MPI) integration. The challenging parallel programming task of generating MPI-based parallel programs has remained unexplored. This study first investigates the performance of state-of-the-art language models in generating MPI-based parallel programs. Findings reveal that widely used models such as GPT-3.5 and PolyCoder (specialized multi-lingual code models) exhibit notable performance degradation, when generating MPI-based programs compared to general-purpose programs. In contrast, domain-specific models such as MonoCoder, which are pretrained on MPI-related programming languages of C and C++, outperform larger models. Subsequently, we introduce a dedicated downstream task of MPI-based program generation by fine-tuning MonoCoder on HPCorpusMPI. We call the resulting model as MPIrigen. We propose an innovative preprocessing for completion only after observing the whole code, thus enabling better completion with a wider context. Comparative analysis against GPT-3.5 zero-shot performance, using a novel HPC-oriented evaluation method, demonstrates that MPIrigen excels in generating accurate MPI functions up to 0.8 accuracy in location and function predictions, and with more than 0.9 accuracy for argument predictions. The success of this tailored solution underscores the importance of domain-specific fine-tuning in optimizing language models for parallel computing code generation, paving the way for a new generation of automatic parallelization tools. The sources of this work are available at our GitHub MPIrigen repository: https://github.com/Scientific-Computing-Lab-NRCN/MPI-rigen

翻译：跨多节点规模化计算的迫切需求凸显了高效并行计算的重要性，尤其是在消息传递接口（MPI）集成领域。然而基于MPI的并行程序生成这一具有挑战性的并行编程任务至今尚未被充分探索。本研究首先探究了现有先进语言模型在生成基于MPI的并行程序时的性能表现。研究发现，GPT-3.5和PolyCoder（专用多语言代码模型）等广泛使用的模型在生成基于MPI的程序时，其性能相较于通用程序出现显著下降。相反，领域特定模型（如MonoCoder）经过C和C++等MPI相关编程语言预训练后，其表现优于更大规模的模型。随后，我们通过针对HPCorpusMPI数据集微调MonoCoder，构建了一个专门的基于MPI程序生成的下游任务，并将所得模型命名为MPIrigen。我们提出了一种创新的预处理方法——仅在观察完整代码后进行补全，从而在更广阔的上下文中实现更优质的补全效果。与GPT-3.5零样本性能的对比分析（采用新型高性能计算导向的评估方法）表明，MPIrigen在生成准确的MPI函数方面表现出色：位置和函数预测准确率可达0.8，参数预测准确率超过0.9。这种定制化方案的成功凸显了领域特定微调在优化面向并行计算代码生成的语言模型中的关键作用，为新一代自动化并行化工具开辟了道路。本工作源代码可在我们的GitHub仓库MPIrigen中获取：https://github.com/Scientific-Computing-Lab-NRCN/MPI-rigen