Policy gradient algorithms for language models optimize the same objective $J(θ) = \mathbb{E}*{τ\sim p*θ(τ)}[R(τ)]$, which has exactly two factors: the trajectory probability $p_θ(τ)$ and the reward $R(τ)$. Every method from REINFORCE to PPO to GRPO and their descendants modifies one or both factors to address a specific failure in the preceding formulation. Existing surveys organize these methods by domain or chronology, which obscures the rationale behind each design choice and the precise location of its intervention within the gradient estimator. This survey revisits the landscape of LLM policy optimization from $J(θ)$ on first principles and uses the trajectory side, induced by $p_θ(τ)$, and the reward side, induced by $R(τ)$, as the two axes along which methods are located. It covers the path from REINFORCE and PPO to GRPO, as well as post-GRPO variants, Agentic RL, and GRPO-OPD. The resulting framework is unified, diagnostic, and extensible: it analyzes methods from a shared objective, identifies which side each method modifies and why, and applies the same trajectory and reward axes across these settings. Across these settings, the framework also exposes compound failures that no single-side fix resolves and that therefore require joint design of the trajectory side and the reward side. The boundary cases and coupled failures identified by this map mark where existing solutions run out and provide a principled starting point for designing the next generation of LLM policy optimization algorithms.

翻译：策略梯度算法优化语言模型的目标函数 $J(θ) = \mathbb{E}_{\tau\sim p_θ(\tau)}[R(\tau)]$，该函数仅包含两个要素：轨迹概率 $p_θ(\tau)$ 与奖励 $R(\tau)$。从REINFORCE到PPO、GRPO及其衍生方法，均通过修改其中一个或两个要素，以解决先前方法中存在的特定缺陷。现有综述按领域或时间顺序组织这些方法，这模糊了每种设计选择背后的原理及其在梯度估计器中的精确干预位置。本综述从第一性原理出发，以 $J(θ)$ 为基点重新审视LLM策略优化全景，将轨迹侧（由 $p_θ(\tau)$ 诱导）和奖励侧（由 $R(\tau)$ 诱导）作为定位方法的两条轴线。研究涵盖从REINFORCE、PPO到GRPO的演进路径，以及后GRPO变体、Agentic RL和GRPO-OPD。由此构建的框架具有统一性、诊断性和可扩展性：它从共享目标分析各类方法，识别每种方法修改的侧及其原因，并在所有场景中应用相同的轨迹与奖励轴线。该框架还揭示了单侧修正确方法无法解决、因此需要轨迹侧与奖励侧联合设计的复合失效模式。该图谱标识的边界情况与耦合失效标志着现有方法的局限，为设计新一代LLM策略优化算法提供了原则性起点。