The success of Large Language Models in mathematical reasoning relies heavily on the generation of diverse and valid solution paths during the rollout phase. However, current rollout techniques face a fundamental trade-off: token-level sampling often yields redundant trajectories that differ only in rephrasing, while embedding-level methods utilizing random noise frequently disrupt semantic consistency. To resolve this, we introduce N-GRPO, a novel exploration strategy integrated into the Group Relative Policy Optimization (GRPO) framework. Rather than relying on token-level sampling or native embedding-level noise, our approach leverages Semantic Neighbor Mixing. This mechanism dynamically constructs input representations by mixing the embeddings of an anchor token and its nearest semantic neighbors, thereby injecting diversity while strictly adhering to the local semantic manifold. Experimental evaluations on the DeepSeek-R1-Distill-Qwen models across different sizes show that N-GRPO not only achieves consistent improvements over strong baselines on math reasoning benchmarks but also exhibits robust generalization capabilities on out-of-distribution tasks.

翻译：摘要：大语言模型在数学推理领域的成功高度依赖于推理阶段生成多样且有效的解题路径。然而，当前推理技术面临根本性权衡：词元级采样常产生仅存在措辞差异的冗余轨迹，而利用随机噪声的嵌入级方法则频繁破坏语义一致性。为解决该问题，我们提出N-GRPO——一种集成至组相对策略优化（GRPO）框架的新型探索策略。该方法摒弃词元级采样或原生嵌入级噪声，转而采用语义邻居混合机制：通过混合锚定词元与其最近语义邻居的嵌入表示动态构建输入表征，在严格遵循局部语义流形的前提下注入多样性。基于不同参数量级DeepSeek-R1-Distill-Qwen模型的实验评估表明，N-GRPO不仅在数学推理基准测试中相较强基线方法取得持续性能提升，且展现出对分布外任务的稳健泛化能力。