Large Language Models (LLMs) have demonstrated exceptional coding capability. However, as another critical component of programming proficiency, the debugging capability of LLMs remains relatively unexplored. Previous evaluations of LLMs' debugging ability are significantly limited by the risk of data leakage, the scale of the dataset, and the variety of tested bugs. To overcome these deficiencies, we introduce `DebugBench', an LLM debugging benchmark consisting of 4,253 instances. It covers four major bug categories and 18 minor types in C++, Java, and Python. To construct DebugBench, we collect code snippets from the LeetCode community, implant bugs into source data with GPT-4, and assure rigorous quality checks. We evaluate two commercial and three open-source models in a zero-shot scenario. We find that (1) while closed-source models like GPT-4 exhibit inferior debugging performance compared to humans, open-source models such as Code Llama fail to attain any pass rate scores; (2) the complexity of debugging notably fluctuates depending on the bug category; (3) incorporating runtime feedback has a clear impact on debugging performance which is not always helpful. As an extension, we also compare LLM debugging and code generation, revealing a strong correlation between them for closed-source models. These findings will benefit the development of LLMs in debugging.

翻译：摘要：大型语言模型（LLMs）已展现出卓越的编码能力。然而，作为编程熟练度的另一关键组成部分，LLMs的调试能力仍相对未被充分探索。以往对LLMs调试能力的评估受到数据泄露风险、数据集规模及测试缺陷类型多样性的显著限制。为克服这些不足，我们提出"DebugBench"——一个包含4,253个实例的LLM调试基准。该基准覆盖C++、Java和Python中的四大主要缺陷类别及18种次要类型。构建DebugBench时，我们从LeetCode社区收集代码片段，利用GPT-4向源代码植入缺陷，并确保严格的质量检查。我们在零样本场景下评估了两款商业模型与三款开源模型。研究发现：（1）尽管GPT-4等闭源模型的调试性能不如人类，但Code Llama等开源模型未能获得任何通过率分数；（2）调试复杂度因缺陷类别不同而显著波动；（3）融入运行时反馈对调试性能有显著影响，但这种影响并非总是有益的。作为延伸，我们还将LLM调试与代码生成进行比较，揭示闭源模型的二者之间存在强相关性。这些发现将有助于LLMs在调试方面的发展。