Current benchmarks for evaluating neural code models focus on only a small subset of programming languages, excluding many popular languages such as Go or Rust. To ameliorate this issue, we present the BabelCode framework for execution-based evaluation of any benchmark in any language. BabelCode enables new investigations into the qualitative performance of models' memory, runtime, and individual test case results. Additionally, we present a new code translation dataset called Translating Python Programming Puzzles (TP3) from the Python Programming Puzzles (Schuster et al. 2021) benchmark that involves translating expert-level python functions to any language. With both BabelCode and the TP3 benchmark, we investigate if balancing the distributions of 14 languages in a training dataset improves a large language model's performance on low-resource languages. Training a model on a balanced corpus results in, on average, 12.34% higher $pass@k$ across all tasks and languages compared to the baseline. We find that this strategy achieves 66.48% better $pass@k$ on low-resource languages at the cost of only a 12.94% decrease to high-resource languages. In our three translation tasks, this strategy yields, on average, 30.77% better low-resource $pass@k$ while having 19.58% worse high-resource $pass@k$.

翻译：当前用于评估神经代码模型的基准测试仅涵盖少数编程语言，排除了许多流行语言如Go或Rust。为解决这一问题，我们提出了BabelCode框架，用于对任意基准测试进行任意语言的基于执行的评估。BabelCode使得对模型内存、运行时及单个测试用例结果的定性性能进行新研究成为可能。此外，我们基于Python编程难题（Schuster等人，2021）基准测试，构建了一个名为Translating Python Programming Puzzles（TP3）的新代码翻译数据集，该数据集涉及将专家级Python函数翻译为任意语言。借助BabelCode和TP3基准测试，我们探究了在训练数据集中平衡14种语言的分布是否能提升大型语言模型在低资源语言上的表现。相较于基线，在平衡语料上训练模型后，所有任务和语言的平均$pass@k$提升了12.34%。我们发现，该策略在低资源语言上实现了66.48%的$pass@k$提升，而高资源语言仅下降12.94%。在我们的三个翻译任务中，该策略平均使低资源语言的$pass@k$提升了30.77%，而高资源语言则下降了19.58%。