Reinforcement Learning with Verifiable Rewards (RLVR) has become a central post-training paradigm for improving the reasoning capabilities of large language models. Yet existing methods share a common blind spot: they optimize policies based on instantaneous group-level or batch-level statistics without ever verifying whether the resulting update actually improved the model. This open-loop design -- updating in isolation at each step, guided only by within-group (batch) reward signals -- means optimization can drift or collapse with no mechanism to detect and correct these failures. We argue that the missing ingredient is policy improvement feedback: the ability to measure and optimize inter-iteration progress directly. To this end, we introduce Policy Improvement Reinforcement Learning (PIRL), a framework that replaces surrogate reward maximization with the explicit objective of maximizing cumulative policy improvement across iterations, and prove this temporal objective is perfectly aligned with maximizing final task performance. Building on PIRL, we propose Policy Improvement Policy Optimization (PIPO), which implements closed-loop optimization through retrospective verification. At each iteration, PIPO evaluates whether the previous update yielded genuine improvement against a sliding-window historical baseline, then actively reinforces beneficial updates and suppresses the harmful ones -- transforming an open-loop process into a self-correcting one. We provide theoretical analysis showing that PIPO performs ascent on the PIRL objective in expectation, and experiments on mathematical reasoning benchmarks demonstrate improved stability and performance over GRPO and its variants.

翻译：可验证奖励强化学习（RLVR）已成为提升大型语言模型推理能力的核心后训练范式。然而，现有方法存在一个共同的盲区：它们基于瞬时群体或批次统计量优化策略，却从未验证当前更新是否真正改进了模型。这种开环设计——在每一步孤立更新，仅受组内（批次）奖励信号引导——意味着优化可能发生漂移或崩溃，而缺乏检测与纠正此类失效的机制。我们认为缺失的关键要素是策略改进反馈：即直接测量并优化跨迭代进步的能力。为此，我们提出策略改进强化学习（PIRL）框架，该框架将替代奖励最大化替换为跨迭代累积策略改进最大化的显式目标，并证明这一时间目标与最大化最终任务性能完美对齐。基于PIRL，我们提出策略改进策略优化（PIPO），通过回溯验证实现闭环优化。在每次迭代中，PIPO评估前一次更新相对于滑动窗口历史基线是否产生真实改进，然后主动强化有益更新并抑制有害更新——将开环过程转化为自纠正过程。理论分析表明PIPO在期望意义上对PIRL目标执行梯度上升，数学推理基准实验显示其较GRPO及其变体具有更优的稳定性与性能。