AI-driven software development has rapidly advanced with the emergence of software development agents that leverage large language models (LLMs) to tackle complex, repository-level software engineering tasks. These agents go beyond just generation of final code; they engage in multi-step reasoning, utilize various tools for code modification and debugging, and interact with execution environments to diagnose and iteratively resolve issues. However, most existing evaluations focus primarily on static analyses of final code outputs, yielding limited insights into the agents' dynamic problem-solving processes. To fill this gap, we conduct an in-depth empirical study on 3,977 solving-phase trajectories and 3,931 testing-phase logs from 8 top-ranked agents evaluated on 500 GitHub issues in the SWE-Bench benchmark. Our exploratory analysis shows that Python execution errors during the issue resolution phase correlate with lower resolution rates and increased reasoning overheads. We have identified the most prevalent errors -- such as ModuleNotFoundError and TypeError -- and highlighted particularly challenging errors like OSError and database-related issues (e.g., IntegrityError) that demand significantly more debugging effort. Furthermore, we have discovered 3 bugs in the SWE-Bench platform that affect benchmark fairness and accuracy; these issues have been reported to and confirmed by the maintainers. To promote transparency and foster future research, we publicly share our datasets and analysis scripts.

翻译：随着利用大型语言模型（LLM）处理复杂仓库级软件工程任务的软件开发代理的出现，AI驱动的软件开发已迅速发展。这些代理不仅限于生成最终代码，还涉及多步推理、利用多种工具进行代码修改与调试，并与执行环境交互以诊断和迭代解决问题。然而，现有评估大多聚焦于对最终代码输出的静态分析，对代理动态问题解决过程的洞察有限。为填补这一空白，我们对来自SWE-Bench基准测试中500个GitHub问题的8个顶级代理进行了深入实证研究，分析了3,977条解决阶段轨迹和3,931条测试阶段日志。探索性分析表明，问题解决阶段的Python执行错误与较低解决率和增加的推理开销相关。我们识别了最常见的错误——如ModuleNotFoundError和TypeError——并重点指出了特别具有挑战性的错误，如OSError和数据库相关问题（例如IntegrityError），这些错误需要显著更多的调试工作量。此外，我们在SWE-Bench平台中发现了3个影响基准测试公平性和准确性的缺陷；这些问题已向维护者报告并获确认。为促进透明度和推动未来研究，我们公开分享了数据集和分析脚本。