Large language models (LLMs) have made significant progress in generating codes from textual prompts. However, existing benchmarks have mainly concentrated on translating English prompts to multilingual codes or have been constrained to very limited natural languages (NLs). These benchmarks have overlooked the vast landscape of massively multilingual NL to multilingual code, leaving a critical gap in the evaluation of multilingual LLMs. In response, we introduce HumanEval-XL, a massively multilingual code generation benchmark specifically crafted to address this deficiency. HumanEval-XL establishes connections between 23 NLs and 12 programming languages (PLs), and comprises of a collection of 22,080 prompts with an average of 8.33 test cases. By ensuring parallel data across multiple NLs and PLs, HumanEval-XL offers a comprehensive evaluation platform for multilingual LLMs, allowing the assessment of the understanding of different NLs. Our work serves as a pioneering step towards filling the void in evaluating NL generalization in the area of multilingual code generation. We make our evaluation code and data publicly available at \url{https://github.com/FloatAI/HumanEval-XL}.

翻译：大型语言模型（LLM）在根据文本提示生成代码方面取得了显著进展。然而，现有基准主要集中于将英语提示翻译为多语言代码，或局限于极少数自然语言（NL）。这些基准忽视了从海量多语言自然语言到多语言代码的广阔场景，留下了多语言LLM评估中的关键空白。为此，我们提出了HumanEval-XL，一个专门针对这一缺陷设计的大规模多语言代码生成基准。HumanEval-XL建立了23种自然语言与12种编程语言之间的关联，包含22,080条提示，平均每条提示配有8.33个测试用例。通过确保跨多语言自然语言和编程语言的并行数据，HumanEval-XL为多语言LLM提供了全面的评估平台，能够评估其对不同自然语言的理解能力。我们的工作填补了多语言代码生成领域自然语言泛化评估的空白，迈出了开创性的一步。我们在公开网址 \url{https://github.com/FloatAI/HumanEval-XL} 上提供了评估代码与数据。