Fine-tuning vision-language-action (VLA) policies for long-horizon manipulation still relies heavily on behavior cloning, which requires costly high-quality demonstrations and keeps policies near the demonstration distribution. Reward models can reduce this dependence by reweighting demonstrations and providing dense supervision for on-robot reinforcement learning (RL), but they must be dense, accurate, and general. Existing methods fall short: task-specific stage-aware models are accurate but require per-task annotations, while general vision-language-model (VLM) reward models are broadly applicable but too coarse for fine-grained long-horizon progress. We introduce RM, a multi-task stage-aware reward model that combines an action-primitive-based stage estimator with a multi-gate Mixture-of-Experts (MMoE) value head to produce dense per-step rewards across manipulation tasks. Building on RM, we further propose SPIRAL (Self-Policy Improvement via Reward-Aligned Learning), an on-policy reward-guided framework that improves VLA policies from cheap autonomous rollouts. On a 10-task benchmark, RM reduces value-estimation MSE by 80% over the strongest baselines; when used in SPIRAL, it improves task success from around 50% to near-perfect performance on Folding Shorts (58% to 100%) and Cleaning Whiteboard (50% to 90%), showing that high-quality dense rewards are key to a stable robot data flywheel. Project website: https://qianzhong-chen.github.io/sarm2.github.io/.

翻译：微调视觉-语言-动作（VLA）策略以执行长周期操控任务仍严重依赖行为克隆，这需要昂贵的高质量示范数据，且使策略局限于示范分布。奖励模型可通过重新加权示范数据并为机器人强化学习提供密集监督来降低这种依赖，但其必须满足密集性、准确性和通用性要求。现有方法存在不足：任务特定阶段感知模型虽准确但需为每项任务标注，而通用视觉语言模型奖励模型适用范围广却难以对细粒度长周期进展进行精确评估。我们提出RM——一种多任务阶段感知奖励模型，该模型结合基于动作基元的阶段估计器与多门控混合专家价值头，可在不同操控任务中生成密集的逐步骤奖励。基于RM，我们进一步提出SPIRAL（通过奖励对齐学习实现策略自我提升），这是一种基于策略的奖励引导框架，能够通过低成本自主轨迹提升VLA策略性能。在10项任务基准测试中，RM相比最强基线将价值估计均方误差降低80%；当用于SPIRAL框架时，其在折叠短裤任务（从58%提升至100%）和白板清洁任务（从50%提升至90%）中将成功率从约50%提升至近乎完美，充分证明高质量密集奖励是实现稳定机器人数据飞轮的关键。项目网站：https://qianzhong-chen.github.io/sarm2.github.io/。