In reinforcement learning with verifiable rewards, a benchmark's verifier supplies the reward. Reward-hacking audits mostly look for runs that pass without doing the task. But a run can do the task, cause a side effect, and pass a verifier that checks only the requested change. Do RL reward verifiers notice such a run? We answer with mutation testing: hand-written edits to a correct run leave the work missing or wrong, or add an unrequested change. A pass is a false positive only if our check finds the end state wrong. Every false positive came from an unrequested change, none from missing or wrong work. WorkArena's form verifier, called once after submission, passed forms with and without an unrequested field in all 36 valid pairs. Called after every step, it passed 7 of 11 runs filling that field last, all text or reference fields. In AppWorld, repeating a run's last write changed nothing in 33 of 48 tasks, but in every task of two templates the tests passed a duplicate write. Three recent RL methods train on one; their reward formulas give the duplicate full reward. One count assertion per template rejected all six duplicates and passed all six correct runs.

翻译：暂无翻译