Current benchmarks for evaluating software engineering agents, such as SWE-Bench Verified, are predominantly derived from GitHub issues and fail to accurately reflect how developers interact with chat-based coding assistants in integrated development environments (IDEs). We posit that this mismatch leads to a systematic overestimation of agent's capabilities in real-world scenarios, especially bug fixing. We introduce a novel benchmarking framework that transforms existing formal benchmarks into realistic user queries through systematic analysis of developer interaction patterns with chat-based agents. Our methodology is flexible and can be easily extended to existing benchmarks. In this paper, we apply our testing framework to SWE-Bench Verified, the TypeScript subset of Multi-SWE-Bench and a private benchmark, SWE-Bench C# and transform formal GitHub issue descriptions into realistic user-style queries based on telemetry analysis of a popular chat-based agent interactions. Our findings reveal that existing benchmarks significantly overestimate agent capabilities for some models by >50% over baseline performance for public benchmarks and ~10-16% for our internal benchmark. This work establishes a new paradigm for evaluating interactive chat-based software engineering agents through benchmark mutation techniques.

翻译：当前用于评估软件工程智能体的基准（如SWE-Bench Verified）主要源自GitHub问题，未能准确反映开发者在集成开发环境（IDE）中如何与基于聊天的编码助手进行交互。我们认为这种不匹配导致了对智能体在现实场景（尤其是缺陷修复）中能力的系统性高估。本文提出一种新颖的基准测试框架，该框架通过对开发者与基于聊天的智能体交互模式进行系统分析，将现有形式化基准转化为真实的用户查询。我们的方法具有灵活性，可轻松扩展至现有基准。在本文中，我们将测试框架应用于SWE-Bench Verified、Multi-SWE-Bench的TypeScript子集以及私有基准SWE-Bench C#，并基于对流行聊天智能体交互的遥测分析，将形式化的GitHub问题描述转化为真实的用户风格查询。研究结果表明：对于某些模型，现有基准在公开基准上对智能体能力的高估程度超过基线性能>50%，在内部基准上约为10-16%。这项工作通过基准变异技术，为评估交互式聊天型软件工程智能体建立了新范式。