Reward models (RMs) are crucial for the training of large language models (LLMs), yet they typically rely on large-scale human-annotated preference pairs. With the widespread deployment of LLMs, in-the-wild interactions have emerged as a rich source of implicit reward signals. This raises the question: Can we develop reward models directly from in-the-wild interactions? In this work, we explore this possibility by adopting WildChat as an interaction source and proposing a pipeline to extract reliable human feedback, yielding 186k high-quality instances for training WildReward via ordinal regression directly on user feedback without preference pairs. Extensive experiments demonstrate that WildReward achieves comparable or even superior performance compared to conventional reward models, with improved calibration and cross-sample consistency. We also observe that WildReward benefits directly from user diversity, where more users yield stronger reward models. Finally, we apply WildReward to online DPO training and observe significant improvements across various tasks. Code and data are released at https://github.com/THU-KEG/WildReward.

翻译：奖励模型（RMs）对于大型语言模型（LLMs）的训练至关重要，但它们通常依赖于大规模人工标注的偏好对。随着LLMs的广泛部署，真实世界交互已成为一个丰富的隐式奖励信号来源。这就引出了一个问题：我们能否直接从真实世界交互中开发奖励模型？在本工作中，我们通过采用WildChat作为交互源，并提出了一个提取可靠人类反馈的流程，探索了这种可能性。该流程产生了18.6万个高质量实例，用于通过序数回归直接在用户反馈上训练WildReward，而无需偏好对。大量实验表明，与传统的奖励模型相比，WildReward取得了相当甚至更优的性能，同时具有改进的校准能力和跨样本一致性。我们还观察到，WildReward直接受益于用户多样性，即用户越多，奖励模型越强。最后，我们将WildReward应用于在线DPO训练，并观察到其在各种任务上的显著改进。代码和数据发布于 https://github.com/THU-KEG/WildReward。