Code understanding and generation have fast become some of the most popular applications of language models (LMs). Nonetheless, research on multilingual aspects of Code-LMs (i.e., LMs for code generation) such as cross-lingual transfer between different programming languages, language-specific data augmentation, and post-hoc LM adaptation, alongside exploitation of data sources other than the original textual content, has been much sparser than for their natural language counterparts. In particular, most mainstream Code-LMs have been pre-trained on source code files alone. In this work, we investigate the prospect of leveraging readily available compiler intermediate representations - shared across programming languages - to improve the multilingual capabilities of Code-LMs and facilitate cross-lingual transfer. To this end, we first compile SLTrans, a parallel dataset consisting of nearly 4M self-contained source code files coupled with respective intermediate representations. Next, starting from various base Code-LMs (ranging in size from 1.1B to 7.3B parameters), we carry out continued causal language modelling training on SLTrans, forcing the Code-LMs to (1) learn the IR language and (2) align the IR constructs with respective constructs of various programming languages. Our resulting models, dubbed IRCoder, display sizeable and consistent gains across a wide variety of code generation tasks and metrics, including prompt robustness, multilingual code completion, code understanding, and instruction following.

翻译：代码理解与生成已迅速成为语言模型最热门的应用领域之一。然而，相较于自然语言处理领域，关于代码语言模型（即用于代码生成的语言模型）在多语言方面的研究——例如不同编程语言间的跨语言迁移、特定语言的数据增强、语言模型的后期适配以及利用原始文本内容以外的数据源——仍相对稀少。特别是，大多数主流代码语言模型仅在源代码文件上进行预训练。本研究探讨利用跨编程语言共有的编译器中间表示来提升代码语言模型的多语言能力并促进跨语言迁移的可行性。为此，我们首先构建了SLTrans并行数据集，包含近400万份自包含源代码文件及其对应的中间表示。继而，以不同基础代码语言模型（参数规模从11亿到73亿不等）为起点，我们在SLTrans上持续进行因果语言模型训练，迫使代码语言模型：（1）学习中间表示语言；（2）将中间表示结构与各类编程语言的对应结构进行对齐。最终得到的模型IRCoder，在包括提示鲁棒性、多语言代码补全、代码理解及指令遵循在内的广泛代码生成任务与评估指标中，均展现出显著且一致的性能提升。