Large language models trained on code have shown great potential to increase productivity of software developers. Several execution-based benchmarks have been proposed to evaluate functional correctness of model-generated code on simple programming problems. Nevertheless, it is expensive to perform the same evaluation on complex real-world projects considering the execution cost. On the contrary, static analysis tools such as linters, which can detect errors without running the program, haven't been well explored for evaluating code generation models. In this work, we propose a static evaluation framework to quantify static errors in Python code completions, by leveraging Abstract Syntax Trees. Compared with execution-based evaluation, our method is not only more efficient, but also applicable to code in the wild. For experiments, we collect code context from open source repos to generate one million function bodies using public models. Our static analysis reveals that Undefined Name and Unused Variable are the most common errors among others made by language models. Through extensive studies, we also show the impact of sampling temperature, model size, and context on static errors in code completions.

翻译：在代码上训练的大型语言模型已展现出提升软件开发人员生产力的巨大潜力。已有多个基于执行的基准测试被提出，用于评估模型在简单编程问题上生成代码的功能正确性。然而，考虑到执行成本，对复杂的实际项目进行相同评估代价高昂。相比之下，静态分析工具（如代码检查工具）可在不运行程序的情况下检测错误，却尚未被充分探索用于评估代码生成模型。在本工作中，我们提出了一种静态评估框架，通过利用抽象语法树来量化Python代码补全中的静态错误。与基于执行的评估相比，我们的方法不仅效率更高，而且适用于现实场景中的代码。在实验中，我们从开源代码仓库中采集代码上下文，使用公开模型生成了100万个函数体。我们的静态分析表明，未定义名称和未使用变量是语言模型生成的代码中最常见的错误。通过广泛研究，我们还展示了采样温度、模型大小及上下文对代码补全中静态错误的影响。