Recent advances in reinforcement learning for code generation have made robust environments essential to prevent reward hacking. As LLMs increasingly serve as evaluators in code-based RL, their ability to detect reward hacking remains understudied. In this paper, we propose a novel taxonomy of reward exploits spanning across 54 categories and introduce TRACE (Testing Reward Anomalies in Code Environments), a synthetically curated and human-verified benchmark containing 517 testing trajectories. Unlike prior work that evaluates reward hack detection in isolated classification scenarios, we contrast these evaluations with a more realistic, contrastive anomaly detection setup on TRACE. Our experiments reveal that models capture reward hacks more effectively in contrastive settings than in isolated classification settings, with GPT-5.2 with highest reasoning mode achieving the best detection rate at 63%, up from 45% in isolated settings on TRACE. Building on this insight, we demonstrate that state-of-the-art models struggle significantly more with semantically contextualized reward hacks compared to syntactically contextualized ones. We further conduct qualitative analyses of model behaviors, as well as ablation studies showing that the ratio of benign to hacked trajectories and analysis cluster sizes substantially impact detection performance. We release the benchmark and evaluation harness to enable the community to expand TRACE and evaluate their models.

翻译：近年来，强化学习在代码生成领域的进展使得构建鲁棒的环境以防止奖励攻击变得至关重要。随着大语言模型越来越多地作为基于代码的强化学习中的评估器，其检测奖励攻击的能力仍未得到充分研究。本文提出了一种涵盖54个类别的奖励攻击新型分类法，并介绍了TRACE（代码环境中的奖励异常测试）——一个包含517条测试轨迹、经人工验证的合成基准数据集。与先前在孤立分类场景中评估奖励攻击检测的工作不同，我们在TRACE上将这些评估与更贴近现实的对比式异常检测设置进行对比。实验结果表明，模型在对比设置中比在孤立分类设置中能更有效地捕捉奖励攻击，其中GPT-5.2在最高推理模式下实现了63%的最佳检测率，较其在TRACE孤立设置中的45%有所提升。基于这一发现，我们证明相较于语法上下文化的奖励攻击，最先进的模型在处理语义上下文化的奖励攻击时面临显著更大的困难。我们进一步对模型行为进行了定性分析，并通过消融实验表明，良性轨迹与攻击轨迹的比例以及分析簇的大小对检测性能有实质性影响。我们发布了该基准数据集及评估工具，以支持研究社区扩展TRACE并评估其模型。