Software Engineering Agents (SWE agents) can autonomously perform development tasks on benchmarks like SWE Bench, but still face challenges when tackling complex and ambiguous real-world tasks. Consequently, SWE agents are often designed to allow interactivity with developers, enabling collaborative problem-solving. To understand how developers collaborate with SWE agents and the barriers they face in such interactions, we observed 19 developers using an in-IDE agent to resolve 33 open issues in repositories to which they had previously contributed. Participants successfully resolved about half of these issues, with those solving issues incrementally having greater success than those using a one-shot approach. Participants who actively collaborated with the agent and iterated on its outputs were also more successful, though they faced challenges in trusting the agent's responses and collaborating on debugging and testing. Our findings suggest that to facilitate successful collaborations, both SWE agents and developers should actively contribute to tasks throughout all stages of the software development process. SWE agents can enable this by challenging and engaging in discussions with developers, rather than being conclusive or sycophantic.
翻译:软件工程智能体(SWE agents)能够在SWE Bench等基准测试中自主完成开发任务,但在处理复杂且模糊的真实世界任务时仍面临挑战。因此,SWE智能体通常被设计为支持与开发者的交互,以实现协作式问题求解。为理解开发者如何与SWE智能体协作以及此类交互中面临的障碍,我们观察了19名开发者使用IDE内嵌智能体解决33个其曾参与贡献的代码库中的开放问题。参与者成功解决了其中约半数问题,且采用渐进式解决策略的开发者比采用一次性方法者取得更高成功率。那些积极与智能体协作并对其输出进行迭代优化的参与者同样更为成功,尽管他们在信任智能体响应、以及协同进行调试与测试方面仍面临挑战。我们的研究表明:为促进成功协作,SWE智能体与开发者都应在软件开发全过程的各阶段积极贡献任务。SWE智能体可通过挑战开发者并参与讨论来实现这一目标,而非仅提供结论性意见或盲目附和。