Over the past few years, Large Language Models of Code (Code LLMs) have started to have a significant impact on programming practice. Code LLMs are also emerging as building blocks for research in programming languages and software engineering. However, Code LLMs produce impressive results on programming languages that are well represented in their training data (e.g., Java, Python, or JavaScript), but struggle with low-resource languages that have limited training data available. Low resource languages include OCaml, Racket, and several others. This paper presents an effective approach for boosting the performance of Code LLMs on low-resource languages using semi-synthetic data. Our approach, MultiPL-T, translates training data from high-resource languages into training data for low-resource languages in the following way. 1) We use a Code LLM to synthesize tests for commented code from a high-resource language, filtering out faulty tests and code with low test coverage. 2) We use a Code LLM to translate Python code to a target low-resource language, and use tests to validate the translation. We apply this approach to generate tens of thousands of validated training items for Julia, Lua, OCaml, R, and Racket. Furthermore, we use an open model (StarCoderBase) with open training data (The Stack), which allows us to decontaminate benchmarks, train models without violating licenses, and run experiments that could not otherwise be done. With MultiPL-T generated data, we present fine-tuned versions of StarCoderBase and Code Llama for Julia, Lua, OCaml, R, and Racket. On established benchmarks (MultiPL-E), these models outperform other open Code LLMs. The MultiPL-T approach is easy to apply to new languages, and is significantly more efficient and effective than alternatives such as training longer.

翻译：过去几年中，代码大语言模型（Code LLMs）已开始对编程实践产生显著影响。这些模型正逐渐成为编程语言和软件工程研究的基础组件。然而，代码大语言模型在训练数据中充分覆盖的编程语言（如Java、Python或JavaScript）上表现优异，但在训练数据有限的低资源语言（包括OCaml、Racket等）上表现欠佳。本文提出了一种利用半合成数据提升低资源语言上代码大语言模型性能的有效方法。我们的方法MultiPL-T通过以下方式将高资源语言的训练数据转化为低资源语言的训练数据：1）使用代码大语言模型为高资源语言的注释代码合成测试用例，过滤存在缺陷的测试用例和测试覆盖率低的代码；2）使用代码大语言模型将Python代码翻译为目标低资源语言，并通过测试验证翻译质量。我们应用此方法为Julia、Lua、OCaml、R和Racket生成了数万个经过验证的训练样本。此外，我们采用开源模型（StarCoderBase）和开放训练数据（The Stack），这使得我们能够对基准进行去污染处理、在不违反许可证的前提下训练模型，以及开展其他方法无法实现的实验。基于MultiPL-T生成的数据，我们分别针对Julia、Lua、OCaml、R和Racket发布了StarCoderBase和Code Llama的微调版本。在权威基准测试（MultiPL-E）中，这些模型的表现优于其他开源代码大语言模型。MultiPL-T方法易于应用于新语言，且比延长训练时间等替代方案显著更高效、更有效。