Reinforcement learning for LLM reasoning has rapidly emerged as a prominent research area, marked by a significant surge in related studies on both algorithmic innovations and practical applications. Despite this progress, several critical challenges remain, including the absence of standardized guidelines for employing RL techniques and a fragmented understanding of their underlying mechanisms. Additionally, inconsistent experimental settings, variations in training data, and differences in model initialization have led to conflicting conclusions, obscuring the key characteristics of these techniques and creating confusion among practitioners when selecting appropriate techniques. This paper systematically reviews widely adopted RL techniques through rigorous reproductions and isolated evaluations within a unified open-source framework. We analyze the internal mechanisms, applicable scenarios, and core principles of each technique through fine-grained experiments, including datasets of varying difficulty, model sizes, and architectures. Based on these insights, we present clear guidelines for selecting RL techniques tailored to specific setups, and provide a reliable roadmap for practitioners navigating the RL for the LLM domain. Finally, we reveal that a minimalist combination of two techniques can unlock the learning capability of critic-free policies using vanilla PPO loss. The results demonstrate that our simple combination consistently improves performance, surpassing strategies like GRPO and DAPO.

翻译：强化学习在大语言模型推理中的应用已迅速成为一个重要的研究领域，相关研究在算法创新和实际应用方面均呈现出显著增长。尽管取得了这些进展，仍存在若干关键挑战，包括缺乏使用强化学习技术的标准化指南，以及对其底层机制的理解尚不完整。此外，不一致的实验设置、训练数据的差异以及模型初始化的不同导致了相互矛盾的结论，掩盖了这些技术的关键特征，并使从业者在选择合适技术时感到困惑。本文通过在一个统一的开源框架内进行严格的复现和隔离评估，系统性地回顾了广泛采用的强化学习技术。我们通过细粒度的实验（包括不同难度、模型规模和架构的数据集）分析了每种技术的内部机制、适用场景和核心原理。基于这些见解，我们提出了针对特定设置选择强化学习技术的清晰指南，并为从业者在大语言模型领域应用强化学习提供了可靠的路线图。最后，我们发现，仅需将两种技术进行极简组合，即可利用标准的PPO损失函数，解锁无需评论家策略的学习能力。结果表明，我们的简单组合能持续提升性能，并超越GRPO和DAPO等策略。