Code agents resolve 65-70% of SWE-bench Verified issues, but Pass@1 cannot tell us why the rest fail, and, as we show, capable-model failures are systematically misdiagnosed without trajectory data. We introduce TRAJEVAL, a training-free decomposition of agent trajectories into reference-patch-aligned search, read, and edit stages, and apply it across 16,758 trajectories spanning three architectures and seven models. The dominant failure of capable models is not localization: 60-69% of failures on SWE-Agent and OpenHands reach and edit the correct functions yet still produce incorrect patches, and the pattern persists for most models on the bash-only LiveSWEAgent. Within this Edit-Quality residual, we identify Coherence Collapse, where the agent reaches correct code and then overwrites or thrashes it, as the largest theme, replicating across SWE-bench Verified and the multilingual PolyBench Verified. In 5 cases, the agent produces a patch bit-identical to the gold reference mid-trajectory and destroys it later; an edit-commit checkpoint recovers all 5 against the SWE-bench Docker harness. A reference-free consensus-driven variant yields a directional +3.0 pp Pass@1 measurement on GPT-5 (p=0.08).
翻译:代码智能体能解决 SWE-bench Verified 中 65-70% 的问题,但 Pass@1 指标无法告诉我们其余失败的原因,而且我们表明,没有轨迹数据时,强模型失败的系统性诊断会被错误解读。我们引入 TRAJEVAL,一种免训练的智能体轨迹分解方法,将其划分为与参考补丁对齐的搜索、读取和编辑阶段,并在涵盖三种架构和七种模型的 16,758 条轨迹上应用。强模型的主要失败不在于定位:SWE-Agent 和 OpenHands 中 60-69% 的失败案例能定位并编辑到正确函数,但仍产生错误补丁,且这一模式在仅使用 bash 的 LiveSWEAgent 上对大多数模型依然成立。在此编辑质量残差中,我们识别出“连贯性崩塌”——智能体到达正确代码后覆盖或频繁修改它——作为最主要主题,该现象在 SWE-bench Verified 和多语言 PolyBench Verified 上复现。在 5 个案例中,智能体在生产中期轨迹中生成与黄金参考补丁完全一致的补丁,但随后将其破坏;一个编辑-提交检查点恢复策略能找回所有 5 个案例,并通过 SWE-bench Docker 测试框架。一种免参考的共识驱动变体在 GPT-5 上产生方向性的 Pass@1 提升 +3.0 个百分点(p=0.08)。