Generative Reward Models (GenRMs) and LLM-as-a-Judge exhibit deceptive alignment by producing correct judgments for incorrect reasons, as they are trained and evaluated to prioritize Outcome Accuracy, which undermines their ability to generalize during RLHF. We introduce Rationale Consistency, a fine-grained metric that quantifies the alignment between the model's reasoning process and human judgment. Our evaluation of frontier models reveals that rationale consistency effectively discriminates among state-of-the-art models and detects deceptive alignment, while outcome accuracy falls short in both respects. To mitigate this gap, we introduce a hybrid signal that combines rationale consistency with outcome accuracy for GenRM training. Our training method achieves state-of-the-art performance on RM-Bench (87.1%) and JudgeBench (82%), surpassing outcome-only baselines by an average of 5%. Using RM during RLHF, our method effectively improves performance as demonstrated on Arena Hard v2, notably yielding a 7% improvement in creative writing tasks. Further analysis confirms that our method escapes the deceptive alignment trap, effectively reversing the decline in rationale consistency observed in outcome-only training.

翻译：生成式奖励模型（GenRMs）和LLM-as-a-Judge表现出欺骗性对齐，即基于错误理由产生正确判断，这是因为其训练和评估均以结果准确性为优先目标，从而削弱了它们在RLHF过程中的泛化能力。我们提出了推理一致性这一细粒度指标，用于量化模型推理过程与人类判断之间的对齐程度。对前沿模型的评估表明，推理一致性能够有效区分最先进的模型并检测欺骗性对齐，而结果准确性在这两方面均存在不足。为弥补这一差距，我们引入了一种结合推理一致性与结果准确性的混合信号用于GenRM训练。我们的训练方法在RM-Bench（87.1%）和JudgeBench（82%）上达到了最先进的性能，平均超越仅基于结果的基线方法5%。在RLHF过程中使用我们的奖励模型时，该方法能有效提升性能，这在Arena Hard v2上的测试中得到验证，尤其在创意写作任务中实现了7%的性能提升。进一步分析证实，我们的方法能够规避欺骗性对齐陷阱，有效逆转了仅基于结果的训练中观察到的推理一致性下降趋势。