We present CRM (Multi-Agent Collaborative Reward Model), a framework that replaces a single black-box reward model with a coordinated team of specialist evaluators to improve robustness and interpretability in RLHF. Conventional reward models struggle to jointly optimize multiple, sometimes conflicting, preference dimensions (e.g., factuality, helpfulness, safety) and offer limited transparency into why a score is assigned. CRM addresses these issues by decomposing preference evaluation into domain-specific agents that each produce partial signals, alongside global evaluators such as ranker-based and embedding-similarity rewards. A centralized aggregator fuses these signals at each timestep, balancing factors like step-wise correctness, multi-agent agreement, and repetition penalties, yielding a single training reward compatible with standard RL pipelines. The policy is optimized with advantage-based updates (e.g., GAE), while a value model regresses to the aggregated reward, enabling multi-perspective reward shaping without requiring additional human annotations beyond those used to train the evaluators. To support training and assessment, we introduce rewardBench, a benchmark and training suite aligned with the collaborative structure of CRM. Together, CRM and rewardBench provide a practical, modular path to more transparent reward modeling and more stable optimization.

翻译：我们提出了CRM（多智能体协作奖励模型），该框架通过一个协调的专家评估器团队替代单一的黑盒奖励模型，以提高RLHF的鲁棒性和可解释性。传统奖励模型难以同时优化多个可能相互冲突的偏好维度（如事实性、帮助性、安全性），且对评分依据的透明度有限。CRM通过将偏好评估分解为特定领域的智能体来解决这些问题，每个智能体产生部分信号，同时结合基于排序器和嵌入相似度奖励等全局评估器。中央聚合器在每个时间步融合这些信号，平衡逐步正确性、多智能体一致性和重复惩罚等因素，最终生成与标准RL流程兼容的单一训练奖励。策略通过基于优势的更新（如GAE）进行优化，而价值模型则回归到聚合奖励，从而在不需额外人工标注的情况下实现多视角奖励塑形（仅需训练评估器时使用的标注）。为支持训练和评估，我们引入了rewardBench——一个与CRM协作结构对齐的基准测试和训练套件。CRM与rewardBench共同为更透明的奖励建模和更稳定的优化提供了实用化、模块化的实现路径。