In recent years, researchers have created and introduced a significant number of various code generation models. As human evaluation of every new model version is unfeasible, the community adopted automatic evaluation metrics such as BLEU to approximate the results of human judgement. These metrics originate from the machine translation domain and it is unclear whether they are applicable for the code generation tasks and how well they agree with the human evaluation on this task. There are also other metrics, CodeBLEU and RUBY, developed to estimate the similarity of code, that take into account the properties of source code. However, for these metrics there are hardly any studies on their agreement with the human evaluation. Despite all that, minimal differences in the metric scores have been used in recent papers to claim superiority of some code generation models over the others. In this paper, we present a study on the applicability of six metrics -- BLEU, ROUGE-L, METEOR, ChrF, CodeBLEU, and RUBY -- for evaluation of code generation models. We conduct a study on two different code generation datasets and use human annotators to assess the quality of all models run on these datasets. The results indicate that for the CoNaLa dataset of Python one-liners, none of the metrics can correctly emulate human judgement on which model is better with >95% certainty if the difference in model scores is less than 5 points. For the HearthStone dataset, which consists of classes of a particular structure, a difference in model scores of at least 2 points is enough to claim the superiority of one model over the other. Our findings suggest that the ChrF metric is a better fit for the evaluation of code generation models than the commonly used BLEU and CodeBLEU. Yet, finding a metric for code generation that closely agrees with humans requires additional work.

翻译：近年来，研究人员创建并引入了大量不同的代码生成模型。由于对每个新模型版本进行人工评估不可行，学术界采用了BLEU等自动评估指标来近似人工评估的结果。这些指标源于机器翻译领域，尚不确定它们是否适用于代码生成任务，以及在该任务中与人工评估的一致性如何。此外，还存在其他指标如CodeBLEU和RUBY，专门用于估计代码相似性，并考虑了源代码的特性。然而，对于这些指标与人工评估一致性的研究几乎空白。尽管如此，近期论文中仍使用指标分数的微小差异来宣称某些代码生成模型优于其他模型。本文针对六项指标——BLEU、ROUGE-L、METEOR、ChrF、CodeBLEU和RUBY——在代码生成模型评估中的适用性进行了研究。我们在两个不同的代码生成数据集上开展研究，并利用人工标注者评估所有模型在这些数据集上的输出质量。结果表明，对于包含Python单行代码的CoNaLa数据集，当模型得分差异小于5分时，没有任何指标能以超过95%的置信度正确模拟人工对模型优劣的判断。而对于由特定结构类组成的HearthStone数据集，模型得分至少相差2分即可宣称一个模型优于另一个模型。我们的发现表明，ChrF指标比常用的BLEU和CodeBLEU更适用于代码生成模型的评估。然而，要找到与人工评估高度一致的代码生成指标仍需进一步研究。