Reward models (RMs) are critical to reinforcement learning from human feedback (RLHF). However, policy models can exploit biases or spurious correlations in dense RMs, causing reward hacking: RM scores rise while alignment with human preferences deteriorates, especially under distribution shift. We propose UMM-RM (Upcycle-and-Merge MoE Reward Model), which copies the feed-forward layers of an SFT checkpoint to initialize shared and routed experts, then trains the model on preference data. The always-active shared expert captures instruction-agnostic preference signals, while routed experts model fine-grained preferences across instructions or tasks. The experts are then merged into a single dense RM using learnable weights, preserving the robustness and exploitation resistance of expert diversity without MoE or ensemble inference overhead. Across multiple base models and preference datasets, UMM-RM improves preference accuracy, reduces reward hacking during PPO training, and achieves more stable preference alignment than standard dense RMs.

翻译：暂无翻译