As large language models (LLMs) increasingly interact with external tools, reward modeling for tool use has emerged as a critical yet underexplored area of research. Existing reward models, trained primarily on natural language outputs, struggle to evaluate tool-based reasoning and execution. To quantify this gap, we introduce FC-RewardBench, the first benchmark to systematically evaluate reward models in tool-calling scenarios. Our analysis shows that current reward models frequently miss key signals of effective tool use, highlighting the need for domain-specific modeling. We address this by proposing a training framework for outcome reward models using data synthesized from permissively licensed, open-weight LLMs. We introduce ToolRM - a suite of reward models for tool-use ranging from 1.7B to 14B parameters. Across diverse settings, these models consistently outperform general-purpose baselines. Notably, they achieve up to a 25% improvement with Best-of-N sampling, while also improving robustness to input noise, enabling effective data filtering, and supporting RL-training of policy models.

翻译：随着大语言模型（LLMs）日益频繁地与外部工具交互，工具使用的奖励建模已成为关键但尚未充分探索的研究领域。现有奖励模型主要基于自然语言输出进行训练，难以有效评估基于工具的推理与执行过程。为量化这一差距，我们提出了FC-RewardBench——首个系统评估工具调用场景下奖励模型的基准测试。分析表明，当前奖励模型经常遗漏有效工具使用的关键信号，凸显了领域专用建模的必要性。针对此问题，我们提出了基于宽松许可开放权重LLMs合成数据的结果奖励模型训练框架，并推出了ToolRM——一套参数量从17亿到140亿不等的工具使用奖励模型。在多样化场景中，这些模型始终优于通用基线模型。值得注意的是，在Best-of-N采样策略下其性能提升最高可达25%，同时增强了输入噪声的鲁棒性，实现了有效数据过滤，并支持策略模型的强化学习训练。