With the advent of DeepSeek-R1, a new wave of reinforcement learning (RL) methods has emerged that seem to unlock stronger mathematical reasoning. However, a closer look at the open-source ecosystem reveals a critical limitation: with sufficiently many draws (e.g., $\texttt{pass@1024}$), many existing base models already solve nearly all questions on widely used math benchmarks such as MATH-500 and AIME 2024. This suggests that the RL fine-tuning methods prevalent in the LLM reasoning literature largely sharpen existing solution modes rather than discovering entirely new ones. Such sharpening stands in contrast to the broader promise of RL: to foster exploration and to acquire new skills. To move beyond this plateau, we introduce MATH-Beyond (MATH-B), a benchmark deliberately constructed to defeat common open-source models of up to 8B parameters even under large sampling budgets. Improving performance on our benchmark via RL requires methods that learn to reason in ways that go beyond base model capabilities in repeated sampling. Since the problems are drawn from subsets of DAPO-Math-17K and DeepScaleR datasets, they remain topically equivalent to standard high-school math. Validating our premise, RL fine-tuned models such as Nemotron-Research-Reasoning-Qwen-1.5B and DeepScaleR-1.5B-Preview perform poorly on MATH-B at $\texttt{pass@1024}$, showing how existing approaches fall short on tackling harder instances. We hope MATH-B will catalyze exploration-driven RL approaches that elicit deeper reasoning capabilities. We release MATH-B at https://huggingface.co/datasets/brendel-group/MATH-Beyond.

翻译：随着DeepSeek-R1的问世，涌现出一波新的强化学习（RL）方法，这些方法似乎能够解锁更强的数学推理能力。然而，仔细观察开源生态系统会发现一个关键局限：在足够多的采样次数下（例如 $\texttt{pass@1024}$），许多现有的基础模型已经能够解决广泛使用的数学基准测试（如MATH-500和AIME 2024）中几乎所有问题。这表明，当前大语言模型推理文献中主流的RL微调方法，主要是对现有解题模式进行锐化，而非发现全新的模式。这种锐化与强化学习更广泛的承诺——促进探索和获取新技能——形成了对比。为了突破这一平台期，我们引入了MATH-Beyond（MATH-B），这是一个专门构建的基准测试，旨在即使在大规模采样预算下，也能击败参数规模高达80亿的常见开源模型。要通过RL方法在我们的基准测试上提升性能，就需要学习方法能够在重复采样中，以超越基础模型固有能力的推理方式进行推理。由于问题选自DAPO-Math-17K和DeepScaleR数据集的子集，它们在主题上仍等同于标准的高中数学。验证我们的前提，经过RL微调的模型，如Nemotron-Research-Reasoning-Qwen-1.5B和DeepScaleR-1.5B-Preview，在 $\texttt{pass@1024}$ 下于MATH-B上表现不佳，这显示了现有方法在处理更困难实例时的不足。我们希望MATH-B能够催化探索驱动的RL方法，从而激发更深层次的推理能力。我们在 https://huggingface.co/datasets/brendel-group/MATH-Beyond 上发布了MATH-B。