Large language models (LLMs) are increasingly trusted as automated judges, assisting evaluation and providing reward signals for training other models, particularly in reference-based settings like Reinforcement Learning with Verifiable Rewards (RLVR). However, we uncover a critical vulnerability even in this reference-based paradigm: generative reward models are systematically susceptible to reward hacking. We find that superficial inputs, which we term ''master keys'' such as non-word symbols (e.g., '':'' or ''.'') or generic reasoning openers (e.g., ''Thought process:'' or ''Let's solve this problem step by step.''), can consistently elicit false positive rewards without any substantive reasoning. Our systematic evaluation demonstrates this is a widespread failure affecting a diverse range of models, including leading proprietary systems such as GPT-o1 and Claude-4. These results challenge the assumed robustness of LLM judges and pose a significant threat to their reliability. To address this, we propose a simple yet effective data augmentation strategy using truncated model outputs as adversarial negative examples. The resulting Master Reward Models (Master-RMs) demonstrate state-of-the-art robustness against these ''master key'' attacks while maintaining high performance in standard evaluation settings. We supplement these findings with a comprehensive analysis of the vulnerability across model scales, prompt variations, and common inference-time strategies, offering insights to guide future research on robust LLM evaluation. We release our robust, general-domain reward models and the synthetic training data at https://huggingface.co/sarosavo/Master-RM and https://huggingface.co/datasets/sarosavo/Master-RM.

翻译：大型语言模型（LLM）正日益被信赖为自动化评判者，协助评估并为训练其他模型提供奖励信号，尤其在基于参考的设置中，例如基于可验证奖励的强化学习（RLVR）。然而，我们揭示出即使在基于参考的范式下也存在一个关键漏洞：生成式奖励模型系统性地易受奖励黑客攻击。我们发现，浅层输入（我们称之为“万能钥匙”），例如非单词符号（如“:”或“.”）或通用推理开场白（如“思考过程：”或“让我们逐步解决这个问题。”），可以在没有任何实质性推理的情况下始终如一地引发假阳性奖励。我们的系统性评估表明，这是一种影响多种模型的普遍失败，包括领先的专有系统，如GPT-o1和Claude-4。这些结果挑战了LLM评判者被认为的鲁棒性，并对其可靠性构成重大威胁。为了解决这一问题，我们提出了一种简单而有效的数据增强策略，即使用截断的模型输出作为对抗性负例。由此产生的Master-RM展现了针对这些“万能钥匙”攻击的最先进的鲁棒性，同时在标准评估设置中保持高性能。我们通过对不同模型规模、提示变体和常见推理时策略的漏洞全面分析来补充这些发现，为未来关于鲁棒LLM评估的研究提供见解。我们在https://huggingface.co/sarosavo/Master-RM和https://huggingface.co/datasets/sarosavo/Master-RM发布了鲁棒、通用领域的奖励模型和合成训练数据。