We propose Process-Aware Policy Optimization (PAPO), a method that integrates process-level evaluation into Group Relative Policy Optimization (GRPO) through decoupled advantage normalization, to address two limitations of existing reward designs. Outcome reward models (ORM) evaluate only final-answer correctness, treating all correct responses identically regardless of reasoning quality, and gradually lose the advantage signal as groups become uniformly correct. Process reward models (PRM) offer richer supervision, but directly using PRM scores causes reward hacking, where models exploit verbosity to inflate scores while accuracy collapses. PAPO resolves both by composing the advantage from an outcome component Aout, derived from ORM and normalized over all responses, and a process component Aproc, derived from a rubric-based PRM and normalized exclusively among correct responses. This decoupled design ensures that Aout anchors training on correctness while Aproc differentiates reasoning quality without distorting the outcome signal. Experiments across multiple model scales and six benchmarks demonstrate that PAPO consistently outperforms ORM, reaching 51.3% vs.\ 46.3% on OlympiadBench while continuing to improve as ORM plateaus and declines.

翻译：我们提出过程感知策略优化（PAPO），该方法通过解耦优势归一化将过程级评估融入组相对策略优化（GRPO），以解决现有奖励设计的两个局限性。结果奖励模型（ORM）仅评估最终答案的正确性，对所有正确响应一视同仁而不考虑推理质量，并且当组内响应逐渐趋于一致正确时会丧失优势信号。过程奖励模型（PRM）提供更丰富的监督信号，但直接使用PRM分数会导致奖励攻击，即模型利用冗长性来抬高分数而准确率却崩溃。PAPO通过将优势分解为两部分来解决这两个问题：结果分量Aout（基于ORM推导并在所有响应上归一化）和过程分量Aproc（基于评分细则的PRM推导并仅在正确响应内部归一化）。这种解耦设计确保Aout锚定正确性训练，而Aproc在区分推理质量的同时不扭曲结果信号。跨多个模型规模和六个基准的实验表明，PAPO一致优于ORM，在OlympiadBench上达到51.3%对46.3%，并且在ORM趋于平稳和下降时仍持续提升。