Large Language Models (LLMs) for code are rapidly evolving, with code editing emerging as a critical capability. We introduce CodeEditorBench, an evaluation framework designed to rigorously assess the performance of LLMs in code editing tasks, including debugging, translating, polishing, and requirement switching. Unlike existing benchmarks focusing solely on code generation, CodeEditorBench emphasizes real-world scenarios and practical aspects of software development. We curate diverse coding challenges and scenarios from five sources, covering various programming languages, complexity levels, and editing tasks. Evaluation of 19 LLMs reveals that closed-source models (particularly Gemini-Ultra and GPT-4), outperform open-source models in CodeEditorBench, highlighting differences in model performance based on problem types and prompt sensitivities. CodeEditorBench aims to catalyze advancements in LLMs by providing a robust platform for assessing code editing capabilities. We will release all prompts and datasets to enable the community to expand the dataset and benchmark emerging LLMs. By introducing CodeEditorBench, we contribute to the advancement of LLMs in code editing and provide a valuable resource for researchers and practitioners.

翻译：大型语言模型（LLMs）在代码领域的应用正迅速发展，其中代码编辑成为一项关键能力。我们提出CodeEditorBench，一个用于严格评估LLM在代码编辑任务（包括调试、翻译、优化和需求切换）中性能的评估框架。与仅关注代码生成的现有基准不同，CodeEditorBench强调软件开发的真实场景和实际方面。我们从五个来源精心收集了多样化的编程挑战和场景，涵盖多种编程语言、复杂度级别和编辑任务。对19个LLM的评估显示，闭源模型（尤其是Gemini-Ultra和GPT-4）在CodeEditorBench中优于开源模型，揭示了基于问题类型和提示敏感性的模型性能差异。CodeEditorBench旨在通过提供一个稳健的代码编辑能力评估平台，推动LLM的进步。我们将发布所有提示和数据集，以便社区扩展数据集并评估新兴LLM。通过引入CodeEditorBench，我们为LLM在代码编辑领域的发展做出贡献，并为研究人员和从业者提供宝贵资源。