Latent reasoning offers a more efficient alternative to explicit reasoning by compressing intermediate reasoning into continuous representations and substantially shortening reasoning chains. However, existing latent reasoning methods mainly focus on supervised learning, and reinforcement learning in latent space remains highly unstable. We study this problem through the lens of Group Relative Policy Optimization (GRPO), and show that directly adapting GRPO to latent reasoning is fundamentally non-trivial: latent reasoning changes both the probability density and the sampling mechanism, causing three coupled bottlenecks: absence of intrinsic latent manifolds, where unconstrained exploration pushes rollouts off the valid latent manifold; exploration-optimization misalignment, where trajectory-level rewards can induce incorrect token-level updates; and latent mixture non-closure, where jointly reinforcing multiple correct latent paths can produce an invalid averaged state. To address them, we propose \textbf{Latent-GRPO}, which combines invalid-sample advantage masking, one-sided noise sampling, and optimal correct-path first-token selection. Across four low-difficulty benchmarks (e.g., GSM8K-Aug) and four high-difficulty benchmarks (e.g., AIME), Latent-GRPO improves over its latent initialization by 7.86 Pass@1 points on low-difficulty tasks and surpasses explicit GRPO by 4.27 points on high-difficulty tasks while using 3--4$\times$ shorter reasoning chains. It also achieves stronger pass@$k$ performance under Gumbel sampling. These results establish Latent-GRPO as an effective approach for stable and efficient latent reasoning.

翻译：潜在推理通过将中间推理过程压缩为连续表示并大幅缩短推理链，提供了一种比显式推理更高效的替代方案。然而，现有潜在推理方法主要集中于监督学习，且潜在空间中的强化学习仍高度不稳定。我们通过组相对策略优化（GRPO）的视角研究该问题，并表明直接将GRPO适配至潜在推理存在本质困难：潜在推理同时改变了概率密度与采样机制，导致三个耦合瓶颈——固有潜在流形缺失（无约束探索导致轨迹偏离有效潜在流形）、探索-优化错配（轨迹级奖励可能引发错误的词元级更新）以及潜在混合非封闭性（联合增强多条正确潜在路径会产生无效的平均状态）。为解决这些问题，我们提出\textbf{Latent-GRPO}，该方法结合了无效样本优势掩码、单侧噪声采样和最优正确路径首词元选择。在四个低难度基准（如GSM8K-Aug）和四个高难度基准（如AIME）上，Latent-GRPO在低难度任务中相较其潜在初始化方法提升7.86个Pass@1百分点，在高难度任务中超越显式GRPO 4.27个百分点，同时使用3-4倍更短的推理链。该方法在Gumbel采样下也取得了更强的Pass@$k$性能。这些结果确立了Latent-GRPO作为实现稳定高效潜在推理的有效方法。