Current benchmarks for evaluating neural code models focus on only a small subset of programming languages, excluding many popular languages such as Go or Rust. To ameliorate this issue, we present the BabelCode framework for execution-based evaluation of any benchmark in any language. BabelCode enables new investigations into the qualitative performance of models' memory, runtime, and individual test case results. Additionally, we present a new code translation dataset called Translating Python Programming Puzzles (TP3) from the Python Programming Puzzles (Schuster et al. 2021) benchmark that involves translating expert-level python functions to any language. With both BabelCode and the TP3 benchmark, we investigate if balancing the distributions of 14 languages in a training dataset improves a large language model's performance on low-resource languages. Training a model on a balanced corpus results in, on average, 12.34% higher $pass@k$ across all tasks and languages compared to the baseline. We find that this strategy achieves 66.48% better $pass@k$ on low-resource languages at the cost of only a 12.94% decrease to high-resource languages. In our three translation tasks, this strategy yields, on average, 30.77% better low-resource $pass@k$ while having 19.58% worse high-resource $pass@k$.

翻译：现有用于评估神经代码模型的基准测试仅涵盖少数编程语言，排除了Go或Rust等众多流行语言。为解决这一问题，我们提出了BabelCode框架，用于对任何基准测试中任意语言进行基于执行的评估。BabelCode支持对模型的内存、运行时间和单个测试用例结果的定性性能进行全新研究。此外，我们还从Python编程谜题（Schuster等人，2021）基准测试中构建了一个名为将Python编程谜题（TP3）的新代码翻译数据集，该数据集涉及将专家级Python函数翻译成任意语言。利用BabelCode和TP3基准测试，我们研究了在训练数据集中平衡14种语言的分布是否能够提升大型语言模型在低资源语言上的性能。与基线相比，在平衡语料库上训练模型使所有任务和语言的$pass@k$平均提高了12.34%。我们发现，这一策略在低资源语言上实现了66.48%的$pass@k$提升，仅以高资源语言降低12.94%为代价。在我们的三个翻译任务中，该策略在低资源$pass@k$上平均提高了30.77%，而高资源$pass@k$则下降了19.58%。