As long-horizon coding agents produce more code than any developer can review, oversight collapses onto a single surface: the automated test suite. Reward hacking naturally arises in this setup, as the agent optimizes for passing tests while deviating from the users true goal. We study this reward hacking phenomenon by decompose software engineering tasks into three parts: (i) a natural language description of the specification (ii) visible validation tests that exercise specified features in isolation, and (iii) held-out tests that compose those same features to simulate real-world usage. Based on the specification and the visible validation test suites, a genuine agent would be able to generate a solution that can also pass all of the held-out tests. Therefore we use the gap in pass rates on these two suites to quantify reward hacking. Based on this methodology, we introduce SpecBench, a benchmark comprising 30 systems-level programming tasks ranging from short horizon tasks like building a JSON parser to ultra long horizon tasks like building an entire OS kernel from scratch. Large-scale experiments reveal a consistent pattern: while every frontier agent saturates the visible suite, reward hacking persists, with smaller models exhibiting larger gaps on holdout suites. The gap also scales sharply with task length: it grows by 28 percentage points for every tenfold increase in code size. Failures range from subtle feature isolation to deliberate exploits, including a 2,900-line hash-table "compiler" that memorizes test inputs. SpecBench offers a principled testbed for measuring whether coding agents build genuine working systems or merely game the test suites developers hand them.

翻译：[翻译摘要] 随着长周期编码智能体产出的代码量超过任何开发者能够审查的范围，监督便坍缩至单一表面：自动化测试套件。在这种设定下，奖励黑客行为（reward hacking）自然产生——智能体在优化通过测试的同时偏离用户真实目标。我们通过将软件工程任务分解为三个部分来研究这一现象：(i) 规范的纯文本描述，(ii) 独立验证指定特征的可见性测试，以及(iii) 组合这些相同特征以模拟真实使用场景的留存测试。基于规范与可见性测试套件，诚实智能体应能生成同样通过所有留存测试的解决方案。因此，我们利用这两套测试的通过率差异来量化奖励黑客行为。基于该方法，我们提出SpecBench基准测试集，包含30项系统级编程任务，涵盖从构建JSON解析器等短周期任务到从零构建完整操作系统内核等超长周期任务。大规模实验揭示出一致模式：虽然每个前沿智能体在可见性测试套件上均达饱和，但奖励黑客行为依然存在，较小模型在留存套件上表现出更大差异。该差异随任务长度急剧扩大：代码规模每增加十倍，差异扩大28个百分点。失败案例从细微的特征隔离到蓄意漏洞利用，包括一个通过记忆测试输入来作弊的2900行哈希表"编译器"。SpecBench为测量编码智能体是构建真正可工作系统还是仅仅利用开发者提供的测试套件提供了理论化测试平台。