Code completion models have made significant progress in recent years, yet current popular evaluation datasets, such as HumanEval and MBPP, predominantly focus on code completion tasks within a single file. This over-simplified setting falls short of representing the real-world software development scenario where repositories span multiple files with numerous cross-file dependencies, and accessing and understanding cross-file context is often required to complete the code correctly. To fill in this gap, we propose CrossCodeEval, a diverse and multilingual code completion benchmark that necessitates an in-depth cross-file contextual understanding to complete the code accurately. CrossCodeEval is built on a diverse set of real-world, open-sourced, permissively-licensed repositories in four popular programming languages: Python, Java, TypeScript, and C#. To create examples that strictly require cross-file context for accurate completion, we propose a straightforward yet efficient static-analysis-based approach to pinpoint the use of cross-file context within the current file. Extensive experiments on state-of-the-art code language models like CodeGen and StarCoder demonstrate that CrossCodeEval is extremely challenging when the relevant cross-file context is absent, and we see clear improvements when adding these context into the prompt. However, despite such improvements, the pinnacle of performance remains notably unattained even with the highest-performing model, indicating that CrossCodeEval is also capable of assessing model's capability in leveraging extensive context to make better code completion. Finally, we benchmarked various methods in retrieving cross-file context, and show that CrossCodeEval can also be used to measure the capability of code retrievers.

翻译：近年来，代码补全模型取得了显著进展，但当前流行的评估数据集（如HumanEval和MBPP）主要聚焦于单文件内的代码补全任务。这种过度简化的设定无法充分反映真实软件开发场景——仓库通常跨多个文件且包含大量跨文件依赖关系，要正确完成代码补全，往往需要获取并理解跨文件的上下文信息。为填补这一空白，我们提出了CrossCodeEval——一个多样化且多语言的代码补全基准测试，要求对跨文件上下文进行深入理解才能准确完成代码补全。CrossCodeEval基于四个主流编程语言（Python、Java、TypeScript和C#）的真实开源且许可证宽松的代码仓库构建。为生成严格依赖跨文件上下文才能准确完成的示例，我们提出了一种基于静态分析的直接高效方法，用于精确定位当前文件中使用的跨文件上下文。在CodeGen和StarCoder等最先进的代码语言模型上进行的大量实验表明，当缺乏相关跨文件上下文时，CrossCodeEval极具挑战性；而将此类上下文加入提示词后，模型性能得到明显提升。然而，即使采用性能最佳的模型，其最终表现仍远未达到理想水平，这表明CrossCodeEval还能有效评估模型利用丰富上下文改善代码补全的能力。最后，我们对多种跨文件上下文检索方法进行了基准测试，结果显示CrossCodeEval还可用于衡量代码检索器的能力。