Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capabilities of Large Language Models (LLMs). However, methods such as GRPO and DAPO suffer from substantial computational cost, since they rely on sampling many rollouts for each prompt. Moreover, in RLVR the relative advantage is often sparse: many samples become nearly all-correct or all-incorrect, yielding low within-group reward variance and thus weak learning signals. In this paper, we introduce arrol (Accelerating RLVR via online Rollout Pruning), an online rollout pruning method that prunes rollouts during generation while explicitly steering the surviving ones more correctness-balanced to enhance learning signals. Specifically, arrol trains a lightweight quality head on-the-fly to predict the success probability of partial rollouts and uses it to make early pruning decisions. The learned quality head can further weigh candidates to improve inference accuracy during test-time scaling. To improve efficiency, we present a system design that prunes rollouts inside the inference engine and re-batches the remaining ones for log-probability computation and policy updates. Across GRPO and DAPO on Qwen-3 and LLaMA-3.2 models (1B-8B), arrol improves average accuracy by +2.30 to +2.99 while achieving up to 1.7x training speedup, and yielding up to +8.33 additional gains in average accuracy in test-time scaling. The code is available at https://github.com/Hsu1023/ARRoL.

翻译：基于可验证奖励的强化学习（RLVR）显著提升了大型语言模型（LLM）的推理能力。然而，GRPO与DAPO等方法因需为每个提示采样大量展开，导致计算成本高昂。此外，RLVR中的相对优势往往稀疏：许多样本近乎全对或全错，组内奖励方差较低，从而削弱学习信号。本文提出arrol（面向RLVR的在线展开剪枝加速），一种在线展开剪枝方法，在生成过程中剪枝展开，同时明确引导幸存展开更具正确性平衡，以增强学习信号。具体而言，arrol在线训练一个轻量级质量头，用于预测部分展开的成功概率，并据此做出早期剪枝决策。所学到的质量头可进一步对候选进行加权，以提升测试时扩展的推理准确性。为提升效率，我们提出一种系统设计，在推理引擎内部剪枝展开，并对剩余展开重新批处理以进行对数概率计算和策略更新。在基于Qwen-3与LLaMA-3.2模型（1B-8B）的GRPO与DAPO实验中，arrol使平均准确率提升+2.30至+2.99，同时实现高达1.7倍的训练加速，并在测试时扩展中额外获得最高+8.33的平均准确率增益。代码已开源：https://github.com/Hsu1023/ARRoL。