Sparse reward reinforcement learning (RL) has become a standard tool for improving LLM reasoning, but its success depends critically on the coverage present in the base model. In practice, models are often primed for RL through \emph{mid-training} on curated reasoning traces that teach useful primitive skills such as decomposition, verification, or self-correction. Although effective, this strategy requires manually specifying what the model should learn, and it remains unclear whether such primitive coverage is enough for much harder problems, which require combining these skills into broader solution strategies. We study a more automated approach: \emph{RL-based mid-training} using large corpora of human-written question-answer data. Rather than treating reference solutions as targets to imitate, our method, ExpRL, uses them as \emph{reward scaffolds}: references are hidden from the policy and used only to construct problem-specific grading rubrics for judging on-policy reasoning traces. The policy samples from the original problem prompt, while an LLM judge compares the sampled reasoning trace against the reference solution and assigns outcome-level or process-level dense rewards. This lets ExpRL reinforce partial progress, useful intermediate reductions, and productive reasoning behaviors that sparse final-answer rewards often fail to upweight. On challenging math reasoning tasks, ExpRL yields stronger RL priming than SFT, sparse-reward GRPO, and self-distillation, and provides a better initialization for subsequent sparse-reward RL. Additional mixed-domain experiments further suggest that ExpRL can extend beyond the original math-only setting.

翻译：稀疏奖励强化学习已成为提升大语言模型推理能力的标准工具，但其成功关键取决于基础模型中的覆盖范围。实践中，通常通过基于人工编排的推理轨迹进行中期训练来为强化学习做准备，这些轨迹教授分解、验证或自我修正等基础技能。尽管有效，但该策略需要手动指定模型应学习的内容，且尚不明确此类基础覆盖是否足以应对需要将这些技能整合为更广泛解决方案的困难问题。我们研究了一种更自动化的方法：使用大规模人工编写的问答数据进行的基于强化学习的中期训练。我们的方法ExpRL并非将参考解答视为模仿目标，而是将其作为奖励支架：隐藏参考解答，仅用于构建针对生成策略推理轨迹的问题特定评分标准。策略从原始问题提示中采样，同时由大语言模型裁判将采样推理轨迹与参考解答进行比对，并分配结果级或过程级密集奖励。这使得ExpRL能够强化稀疏最终答案奖励难以有效提升的部分进展、有用的中间化简以及富有成效的推理行为。在具有挑战性的数学推理任务上，ExpRL相比有监督微调、稀疏奖励GRPO和自蒸馏方法展现出更强的强化学习启动效果，并为后续稀疏奖励强化学习提供了更优的初始化。额外的跨领域实验进一步表明，ExpRL可扩展至原始纯数学场景之外。