Code generation models have increasingly become integral to aiding software development, offering assistance in tasks such as code completion, debugging, and code translation. Although current research has thoroughly examined the correctness of code produced by code generation models, a vital aspect, i.e., the efficiency of the generated code, has often been neglected. This paper presents EffiBench, a benchmark with 1,000 efficiency-critical coding problems for assessing the efficiency of code generated by code generation models. EffiBench contains a diverse set of LeetCode coding problems. Each problem is paired with an executable human-written canonical solution. With EffiBench, we empirically examine the capability of 21 Large Language Models (13 open-sourced and 8 closed-sourced) in generating efficient code. The results demonstrate that GPT-4-turbo generates the most efficient code, significantly outperforming Palm-2-chat-bison, Claude-instant-1, Gemini-pro, GPT-4, and GPT-3.5. Nevertheless, its code efficiency is still worse than the efficiency of human-written canonical solutions. In particular, the average and worst execution time of GPT-4-turbo generated code is 1.69 and 45.49 times that of the canonical solutions.

翻译：代码生成模型已日益成为辅助软件开发的关键工具，在代码补全、调试及代码翻译等任务中提供支持。尽管当前研究已深入探讨了代码生成模型所生成代码的正确性，但一个关键方面——即生成代码的效率——常被忽视。本文提出EffiBench，一个包含1000个效率关键型编程问题的基准测试，用于评估代码生成模型所生成代码的效率。EffiBench涵盖多样化的LeetCode编程问题，每个问题均配有一个可执行的人工编写规范解。借助EffiBench，我们实证检验了21个大语言模型（13个开源模型与8个闭源模型）生成高效代码的能力。结果表明，GPT-4-turbo生成的代码效率最高，显著优于Palm-2-chat-bison、Claude-instant-1、Gemini-pro、GPT-4及GPT-3.5。然而，其代码效率仍低于人工编写的规范解。具体而言，GPT-4-turbo生成代码的平均执行时间与最差执行时间分别为规范解的1.69倍和45.49倍。