Reward modeling is central to alignment pipelines such as RLHF, RLAIF, and PPO-based policy optimization, yet its reliability is constrained by limited and heterogeneous human preference data that are expensive to collect at scale. While synthetic augmentation can expand preference supervision, existing methods often augment uniformly or at the representation level, without targeting examples where the reward model is uncertain or prone to mis-ranking. In this paper, we introduce MARS (Margin and Semantic-Aware Data Augmentation for Reward Modeling), an adaptive augmentation framework that prioritizes low-margin preference pairs and uses semantic distance as a second layer for refinement to enhance the contrast between the chosen and rejected responses. Across multiple preference datasets, reward-model backbones, downstream alignment settings, and benchmarks including RewardBench and AlpacaEval, MARS improves both reward-model quality and alignment performance over existing baselines. Our results show that reward-model augmentation is most effective when guided by both model margins and semantic structure.

翻译：奖励建模是RLHF、RLAIF及基于PPO的策略优化等对齐流程的核心环节，但其可靠性受限于有限且异质性的人类偏好数据（大规模采集成本高昂）。尽管合成数据增强可扩展偏好监督信号，现有方法通常采用均匀增强或基于表征层的增强策略，未能针对奖励模型存在不确定性或易产生误排序的样本进行优化。本文提出MARS（Margin and Semantic-Aware Data Augmentation for Reward Modeling）自适应增强框架，该框架优先处理低边际偏好对，并将语义距离作为第二层精炼指标，以强化被选响应与拒绝响应之间的对比。在多个偏好数据集、奖励模型骨干网络、下游对齐设置及RewardBench、AlpacaEval等基准测试中，MARS在奖励模型质量与对齐性能方面均优于现有基线方法。实验结果表明，当同时基于模型边际与语义结构进行引导时，奖励模型增强能实现最佳效果。