Over the past few years, Large Language Models of Code (Code LLMs) have started to have a significant impact on programming practice. Code LLMs are also emerging as building blocks for research in programming languages and software engineering. However, Code LLMs produce impressive results on programming languages that are well represented in their training data (e.g., Java, Python, or JavaScript), but struggle with low-resource languages that have limited training data available. Low resource languages include OCaml, Racket, and several others. This paper presents an effective approach for boosting the performance of Code LLMs on low-resource languages using semi-synthetic data. Our approach, MultiPL-T, translates training data from high-resource languages into training data for low-resource languages in the following way. 1) We use a Code LLM to synthesize tests for commented code from a high-resource language, filtering out faulty tests and code with low test coverage. 2) We use a Code LLM to translate Python code to a target low-resource language, and use tests to validate the translation. We apply this approach to generate tens of thousands of validated training items for Julia, Lua, OCaml, R, and Racket. Furthermore, we use an open model (StarCoderBase) with open training data (The Stack), which allows us to decontaminate benchmarks, train models without violating licenses, and run experiments that could not otherwise be done. With MultiPL-T generated data, we present fine-tuned versions of StarCoderBase and Code Llama for Julia, Lua, OCaml, R, and Racket. On established benchmarks (MultiPL-E), these models outperform other open Code LLMs. The MultiPL-T approach is easy to apply to new languages, and is significantly more efficient and effective than alternatives such as training longer.

翻译：过去几年，代码大语言模型（Code LLMs）已对编程实践产生深远影响，并逐步成为编程语言与软件工程研究的基础组件。然而，这类模型在训练数据丰富的编程语言（如Java、Python或JavaScript）上表现优异，但在训练数据稀缺的低资源语言（包括OCaml、Racket等）上则面临挑战。本文提出一种利用半合成数据提升代码大语言模型在低资源语言上性能的有效方法——MultiPL-T，其核心思想是将高资源语言的训练数据转化为低资源语言的训练数据，具体流程如下：1）利用代码大语言模型为高资源语言的注释代码合成测试用例，过滤存在缺陷的测试及测试覆盖率低的代码；2）通过代码大语言模型将Python代码翻译至目标低资源语言，并借助测试用例验证翻译质量。我们使用该方法为Julia、Lua、OCaml、R和Racket生成了数万个经过验证的训练样本。此外，我们采用开放模型（StarCoderBase）与开放训练数据（The Stack）进行实验，从而能够对基准测试进行数据污染清除、在无许可违规条件下训练模型，并开展常规方法无法完成的实验。基于MultiPL-T生成的数据，我们针对Julia、Lua、OCaml、R和Racket五种语言微调了StarCoderBase与Code Llama模型。在现有基准测试（MultiPL-E）中，这些模型的表现优于其他开放代码大语言模型。该方法的优势在于易于推广至新语言，且相较于延长训练时长等替代方案，其效率和效果均显著更优。