Recent advances in agentic Reinforcement Learning (RL) have substantially improved the multi-turn tool-use capabilities of large language model agents. However, most existing methods assign credit over coarse heuristic units, such as tool-call boundaries or fixed workflows, making it difficult to identify which intermediate decisions influence downstream outcomes. In this work, we study agentic RL from two perspectives: \textit{where to branch and how to assign credit after branching}. Our pilot analysis shows that influential decision points are broadly distributed throughout the generated sequence rather than concentrated at tool calls, while token entropy alone does not reliably reflect their impact on final outcomes. Motivated by these observations, we propose \textbf{Agentic Procedural Policy Optimization (APPO)}, which shifts branching and credit assignment from coarse interaction units to fine-grained decision points in the sequence. APPO selects branching locations using a Branching Score that combines token uncertainty with policy-induced likelihood gains of subsequent continuations, enabling more targeted exploration while filtering out spurious high-entropy positions. It further introduces procedure-level advantage scaling to better distribute credit across branched rollouts. Experiments on 13 benchmarks show that APPO consistently improves strong agentic RL baselines by nearly 4 points, while keeping efficient tool-calls and maintaining behavior interpretability.

翻译：近期智能体强化学习（RL）的进展显著提升了大型语言模型智能体的多轮工具使用能力。然而，现有方法多基于粗粒度的启发式单元（如工具调用边界或固定工作流）进行信用分配，难以识别影响下游结果的中间决策。本研究从两个视角探讨智能体强化学习：\textit{何处分支，以及分支后如何分配信用}。初步分析表明，影响性决策点广泛分布于生成序列中，而非集中于工具调用处；同时，词元熵值本身无法可靠反映其对最终结果的影响。基于此观察，我们提出\textbf{智能体过程策略优化（APPO）}，将分支与信用分配从粗粒度交互单元迁移至序列中的细粒度决策点。APPO采用结合词元不确定性及后续延续策略诱导似然增益的"分支评分"选取分支位置，在过滤虚假高熵位置的同时实现更具针对性的探索。此外，该方法引入过程级优势缩放机制，以更合理地在分支轨迹间分配信用。13项基准实验表明，APPO在保持高效工具调用与行为可解释性的前提下，将强智能体RL基线一致提升近4个百分点。