Improving the multi-step reasoning ability of Large Language Models (LLMs) is a critical yet challenging task. The dominant paradigm, outcome-supervised reinforcement learning (RLVR), rewards only correct final answers, often propagating flawed reasoning and suffering from sparse reward signals. While process-level reward models (PRMs) provide denser, step-by-step feedback, they lack generalizability and interpretability, requiring task-specific segmentation of the reasoning process. To this end, we propose the Dimension-level Reward Model (DRM), a new supervision framework that bridges the gap between these two approaches. DRM evaluates the quality of a reasoning process along three fundamental, complementary, and interpretable dimensions: Confidence for uncertainty calibration, Relevance for semantic alignment, and Coherence for logical consistency. Together, these dimensions capture aspects beyond final answer correctness and enable interpretable assessment without requiring ground truth answers. Experimental results show that DRM provides effective supervision signals, guides the optimization of LLMs and enhances their reasoning ability. In particular, DRM-supervised training achieves consistent gains on both in-distribution and out-of-distribution open-domain tasks, including mathematics, question answering, code execution, and puzzles. Our findings demonstrate that multidimensional supervision of the reasoning process can improve the generalized reasoning ability of LLMs beyond the training distribution.

翻译：提升大语言模型（LLMs）的多步推理能力是一项关键而具有挑战性的任务。当前的主流范式——结果监督的强化学习（RLVR）——仅奖励正确的最终答案，这常常会传播有缺陷的推理过程并受限于稀疏的奖励信号。虽然过程级奖励模型（PRMs）提供了更密集的、逐步的反馈，但它们缺乏通用性和可解释性，且需要对推理过程进行任务特定的分割。为此，我们提出了维度级奖励模型（DRM），这是一种新的监督框架，旨在弥合这两种方法之间的差距。DRM从三个基础、互补且可解释的维度评估推理过程的质量：用于不确定性校准的置信度、用于语义对齐的相关性以及用于逻辑一致性的连贯性。这些维度共同捕捉了超越最终答案正确性的多个方面，并能在无需真实答案的情况下实现可解释的评估。实验结果表明，DRM能提供有效的监督信号，指导LLMs的优化并增强其推理能力。具体而言，在DRM监督下的训练，在包括数学、问答、代码执行和谜题在内的分布内和分布外开放领域任务上均取得了稳定的性能提升。我们的研究结果表明，对推理过程进行多维监督能够提升LLMs在训练分布之外的泛化推理能力。