Large Language Models (LLMs) have demonstrated exceptional coding capability. However, as another critical component of programming proficiency, the debugging capability of LLMs remains relatively unexplored. Previous evaluations of LLMs' debugging ability are significantly limited by the risk of data leakage, the scale of the dataset, and the variety of tested bugs. To overcome these deficiencies, we introduce `DebugBench', an LLM debugging benchmark consisting of 4,253 instances. It covers four major bug categories and 18 minor types in C++, Java, and Python. To construct DebugBench, we collect code snippets from the LeetCode community, implant bugs into source data with GPT-4, and assure rigorous quality checks. We evaluate two commercial and three open-source models in a zero-shot scenario. We find that (1) while closed-source models like GPT-4 exhibit inferior debugging performance compared to humans, open-source models such as Code Llama fail to attain any pass rate scores; (2) the complexity of debugging notably fluctuates depending on the bug category; (3) incorporating runtime feedback has a clear impact on debugging performance which is not always helpful. As an extension, we also compare LLM debugging and code generation, revealing a strong correlation between them for closed-source models. These findings will benefit the development of LLMs in debugging.

翻译：大语言模型（LLMs）已展现出卓越的编码能力。然而，作为编程熟练度的另一关键组成部分，LLMs的调试能力仍相对未探索。先前对LLMs调试能力的评估因数据泄露风险、数据集规模以及测试的缺陷种类有限而受到显著限制。为弥补这些不足，我们引入了`DebugBench`，一个包含4,253个实例的LLM调试基准测试。它涵盖了C++、Java和Python中的四大主要缺陷类别及18个子类型。为构建DebugBench，我们从LeetCode社区收集代码片段，利用GPT-4在源数据中植入缺陷，并确保严格的质量检查。我们在零样本场景下评估了两款商业模型和三款开源模型。我们发现：(1) 尽管GPT-4等闭源模型的调试性能不及人类，但Code Llama等开源模型未能取得任何通过率得分；(2) 调试的复杂性因缺陷类别不同而显著波动；(3) 引入运行时反馈对调试性能有明显影响，但并非总是有益。作为延伸，我们还比较了LLM调试与代码生成，揭示闭源模型中两者之间存在强相关性。这些发现将有益于LLMs在调试方面的发展。