LLM-based agent systems are emerging as a new software paradigm and have been widely adopted across diverse domains such as medicine, robotics, and programming. However, maintaining these systems requires substantial effort, as they are inevitably prone to bugs and continually evolve to meet changing external requirements. Therefore, automatically resolving agent issues (i.e., bug reports or feature requests) is a crucial and challenging task. While recent software engineering (SE) agents (e.g., SWE-agent) have shown promise in addressing issues in traditional software systems, it remains unclear how effectively they can resolve real-world issues in agent systems, which differ significantly from traditional software. To fill this gap, we first manually analyze 201 real-world agent issues and identify common categories of agent issues. We then spend 500 person-hours constructing AGENTISSUE-BENCH, a reproducible benchmark comprising 50 agent issue resolution tasks (each with an executable environment and failure-triggering tests). We further evaluate state-of-the-art SE agents on AGENTISSUE-BENCH and reveal their limited effectiveness (i.e., with only 3.33% - 12.67% resolution rates). These results underscore the unique challenges of maintaining agent systems compared to traditional software, highlighting the need for further research to develop advanced SE agents for resolving agent issues. Data and code are available at https://alfin06.github.io/AgentIssue-Bench-Leaderboard/#/ .

翻译：基于大语言模型（LLM）的智能体系统正作为一种新兴的软件范式出现，并已在医学、机器人和编程等不同领域得到广泛应用。然而，维护这些系统需要付出巨大努力，因为它们不可避免地容易出现缺陷，并且需要持续演进以满足不断变化的外部需求。因此，自动解决智能体问题（即错误报告或功能请求）是一项至关重要且具有挑战性的任务。尽管近期的软件工程（SE）智能体（例如SWE-agent）在解决传统软件系统的问题方面展现出潜力，但它们在多大程度上能有效解决与现实世界智能体系统相关的问题仍不明确，而这类系统与传统软件存在显著差异。为填补这一空白，我们首先手动分析了201个现实世界中的智能体问题，并识别出智能体问题的常见类别。随后，我们投入500人时构建了AGENTISSUE-BENCH，这是一个可复现的基准测试集，包含50项智能体问题解决任务（每项任务均配有可执行环境及可触发失败的测试）。我们进一步在AGENTISSUE-BENCH上评估了最先进的软件工程智能体，并揭示了其有限的有效性（即仅获得3.33%至12.67%的解决率）。这些结果突显了维护智能体系统相较于传统软件所面临的独特挑战，强调了需要进一步研究以开发更先进的软件工程智能体来解决智能体问题。数据和代码可在 https://alfin06.github.io/AgentIssue-Bench-Leaderboard/#/ 获取。