Recent advances in Large Language Models (LLMs) have underscored the potential of Reinforcement Learning (RL) to facilitate the emergence of reasoning capabilities. Despite the encouraging results, a fundamental dilemma persists as RL improvement relies on learning from high-quality samples, yet the exploration for such samples remains bounded by the inherent limitations of LLMs. This, in effect, creates an undesirable cycle in which what cannot be explored cannot be learned. In this work, we propose Rubric-Scaffolded Reinforcement Learning (RuscaRL), a novel instructional scaffolding framework designed to break the exploration bottleneck for general LLM reasoning. Specifically, RuscaRL introduces checklist-style rubrics as (1) explicit scaffolding for exploration during rollout generation, where different rubrics are provided as external guidance within task instructions to steer diverse high-quality responses. This guidance is gradually decayed over time, encouraging the model to internalize the underlying reasoning patterns; (2) verifiable rewards for exploitation during model training, where we can obtain robust LLM-as-a-Judge scores using rubrics as references, enabling effective RL on general reasoning tasks. Extensive experiments demonstrate the superiority of the proposed RuscaRL across various benchmarks, effectively expanding reasoning boundaries under the Best-of-N evaluation. Our code is available at https://github.com/IANNXANG/RuscaRL.

翻译：近年来，大语言模型（LLMs）的进展凸显了强化学习（RL）在促进推理能力涌现方面的潜力。尽管取得了令人鼓舞的成果，但一个根本困境依然存在：RL的改进依赖于从高质量样本中学习，而对此类样本的探索仍受限于LLMs固有的能力边界。这实际上形成了一个不良循环：无法探索的内容就无法被学习。在本研究中，我们提出了基于量规支架的强化学习（RuscaRL），这是一种新颖的教学支架框架，旨在为通用LLM推理打破探索瓶颈。具体而言，RuscaRL引入清单式量规作为（1）在生成阶段为探索提供显式支架，即在任务指令中提供不同的量规作为外部引导，以引导生成多样化的高质量响应。这种引导会随时间逐渐衰减，鼓励模型内化底层的推理模式；（2）在模型训练阶段为利用提供可验证的奖励，即我们可以使用量规作为参考获得稳健的“LLM即评判者”分数，从而在通用推理任务上实现有效的RL。大量实验证明了所提出的RuscaRL在多种基准测试上的优越性，有效扩展了在Best-of-N评估下的推理边界。我们的代码可在 https://github.com/IANNXANG/RuscaRL 获取。