Effective exploration is a key challenge in reinforcement learning for large language models: discovering high-quality trajectories within a limited sampling budget from the vast natural language sequence space. Existing methods face notable limitations: GRPO samples exclusively from the root, saturating high-probability trajectories while leaving deep, error-prone states under-explored. Tree-based methods blindly disperse budgets across trivial or unrecoverable states, causing sampling dilution that fails to uncover rare correct suffixes and destabilizes local baselines. To address this, we propose Deep Dense Exploration (DDE), a strategy that focuses exploration on $\textit{pivots}$-deep, recoverable states within unsuccessful trajectories. We instantiate DDE with DEEP-GRPO, which introduces three key innovations: (1) a lightweight data-driven utility function that automatically balances recoverability and depth bias to identify pivot states; (2) local dense resampling at each pivot to increase the probability of discovering correct subsequent trajectories; and (3) a dual-stream optimization objective that decouples global policy learning from local corrective updates. Experiments on mathematical reasoning benchmarks demonstrate that our method consistently outperforms GRPO, tree-based methods, and other strong baselines. Code is available at https://github.com/AgentCombo/DEEP-GRPO

翻译：有效探索是大语言模型强化学习中的关键挑战：在有限采样预算内从庞大的自然语言序列空间中发掘高质量轨迹。现有方法存在明显局限：GRPO仅从根节点采样，使高概率轨迹饱和，却导致深层、易错状态探索不足；基于树的方法将预算盲目分散至无关或不可恢复状态，引发采样稀释，既难以发现罕见正确后缀，又破坏局部基线稳定性。为此，我们提出深度稠密探索策略（Deep Dense Exploration, DDE），该策略将探索聚焦于不成功轨迹中的"关键点" —— 深层、可恢复状态。我们通过DEEP-GRPO实例化DDE，该方法包含三项核心创新：（1）轻量级数据驱动效用函数，自动平衡可恢复性与深度偏差以识别关键点状态；（2）对每个关键点进行局部稠密重采样，提升发现后续正确轨迹的概率；（3）双流优化目标，解耦全局策略学习与局部修正更新。数学推理基准实验表明，本方法一致优于GRPO、基于树的方法及其他强基线。代码开源于 https://github.com/AgentCombo/DEEP-GRPO