A well-designed reward is critical for effective reinforcement learning-based policy improvement. In real-world robotics, obtaining such rewards typically requires either labor-intensive human labeling or brittle, handcrafted objectives. Vision-language models (VLMs) have shown promise as automatic reward models, yet their effectiveness on real robot tasks is poorly understood. In this work, we aim to close this gap by introducing (1) RoboReward, a robotics reward dataset and benchmark built on large-scale real-robot corpora from Open X-Embodiment (OXE) and RoboArena, and (2) vision-language reward models trained on this dataset (RoboReward 4B/8B). Because OXE is success-heavy and lacks failure examples, we propose a negative examples data augmentation pipeline that generates calibrated negative and near-misses via counterfactual relabeling of successful episodes and temporal clipping to create partial-progress outcomes from the same videos. Using this framework, we build a large training and evaluation dataset spanning diverse tasks and embodiments to test whether state-of-the-art VLMs can reliably provide rewards for robot learning. Our evaluation of open and proprietary VLMs finds that no model excels across tasks, highlighting substantial room for improvement. We then train general-purpose 4B- and 8B-parameter models that outperform much larger VLMs in assigning rewards for short-horizon robotic tasks. Finally, we deploy the 8B model in real-robot reinforcement learning and find that it improves policy learning over Gemini Robotics-ER 1.5 while narrowing the gap to RL training with human-provided rewards. We release the full dataset, trained reward models, and evaluation suite on our website to advance the development of general-purpose reward models in robotics: https://crfm.stanford.edu/helm/robo-reward-bench (project website).

翻译：设计良好的奖励对于基于强化学习的策略改进至关重要。在实际机器人应用中，获取此类奖励通常需要耗费大量人力进行标注，或依赖脆弱的手工设计目标。视觉-语言模型（VLMs）已展现出作为自动奖励模型的潜力，但其在真实机器人任务中的有效性尚未得到充分验证。本研究旨在通过以下工作填补这一空白：（1）基于Open X-Embodiment（OXE）和RoboArena的大规模真实机器人数据集构建机器人奖励数据集与基准测试平台RoboReward；（2）基于该数据集训练视觉-语言奖励模型（RoboReward 4B/8B）。针对OXE数据集成功案例密集但缺乏失败样本的问题，我们提出负样本数据增强流程：通过对成功轨迹进行反事实重标注生成校准的负样本与接近成功的案例，并利用时序截取技术从相同视频中生成部分进展结果。基于该框架，我们构建了涵盖多样化任务与机器人形态的大规模训练与评估数据集，以验证当前最先进的VLMs能否为机器人学习提供可靠的奖励信号。对开源与专有VLMs的评估表明，现有模型均无法在所有任务中表现优异，这揭示了巨大的改进空间。随后我们训练了通用型4B与80亿参数模型，其在短时程机器人任务的奖励分配中优于规模更大的VLMs。最终，我们将80亿参数模型部署于真实机器人强化学习环境，发现其策略学习效果优于Gemini Robotics-ER 1.5，同时缩小了与人工奖励强化学习训练的差距。为推进机器人通用奖励模型的发展，我们在项目网站发布完整数据集、训练后的奖励模型及评估套件：https://crfm.stanford.edu/helm/robo-reward-bench（项目网站）。