Reward models play a pivotal role in reinforcement learning (RL) and multi-modal trajectory selection for autonomous driving. However, acquiring such rewards typically relies on hand-crafted rule-based objectives or perception ground truth, which hinders generalization for data-scaling. While Vision-Language Models (VLMs) have demonstrated feasibility as reward models in other domains, their effectiveness in driving tasks remains underexplored. In this work, we bridge this gap by (1) introducing DriveReward, a reasoning trajectory evaluation dataset rigorously labeled via temporally-grounded visual guidance, and augmented with counterfactual driving behaviors., (2) alongside a specialized Vision-Language Reward Model. To address the scarcity of failure cases in conventional datasets, we propose a counterfactual data annotation scheme to construct cases encompassing diverse driving styles and erroneous behaviors. Evaluations on our proposed benchmark reveal that even leading open-source and proprietary VLMs fail to excel across all tasks, highlighting significant room for improvement in existing models. Building on these findings, we subsequently tailor a specialized 1B reward model that outperforms larger VLMs on task-specific reward alignment. Finally, we validate our reward model's effectiveness by integrating it into RL finetuning and multi-modal trajectory scoring across multiple baselines, achieving performance comparable to rule-based reward calculations in both open-loop and closed-loop evaluation.

翻译：奖励模型在强化学习及自动驾驶多模态轨迹选择中扮演关键角色。然而，获取此类奖励通常依赖基于手工规则的优化目标或感知真值，这阻碍了数据规模化下的泛化能力。尽管视觉语言模型已在其他领域展现出作为奖励模型的可行性，但其在驾驶任务中的有效性尚未得到充分探索。本研究通过以下工作填补这一空白：（1）提出DriveReward——一个基于时间锚定视觉引导进行严格标注、并辅以反事实驾驶行为的推理轨迹评估数据集；（2）同时设计专用视觉语言奖励模型。针对传统数据集中故障案例匮乏的问题，我们提出反事实数据标注方案，构建涵盖多样化驾驶风格与错误行为的案例。在我们提出的基准测试上的评估表明，即便是领先的开源及商业视觉语言模型也未能全面超越所有任务，凸显现有模型仍有显著改进空间。基于此，我们进一步定制了专用的10亿参数奖励模型，其在任务特定奖励对齐上超越更大规模的视觉语言模型。最后，我们通过将该奖励模型集成到强化学习微调与多模态轨迹评分中，在多种基线上验证其有效性，在开环与闭环评估中均取得了与基于规则的奖励计算相当的性能。