LLM agents have emerged as powerful systems for tackling multi-turn tasks by interleaving internal reasoning and external tool interactions. Agentic Reinforcement Learning has recently drawn significant research attention as a critical post-training paradigm to further refine these capabilities. In this paper, we present AT$^2$PO (Agentic Turn-based Policy Optimization via Tree Search), a unified framework for multi-turn agentic RL that addresses three core challenges: limited exploration diversity, sparse credit assignment, and misaligned policy optimization. AT$^2$PO introduces a turn-level tree structure that jointly enables Entropy-Guided Tree Expansion for strategic exploration and Turn-wise Credit Assignment for fine-grained reward propagation from sparse outcomes. Complementing this, we propose Agentic Turn-based Policy Optimization, a turn-level learning objective that aligns policy updates with the natural decision granularity of agentic interactions. ATPO is orthogonal to tree search and can be readily integrated into any multi-turn RL pipeline. Experiments across seven benchmarks demonstrate consistent improvements over the state-of-the-art baseline by up to 1.84 percentage points in average, with ablation studies validating the effectiveness of each component. Our code is available at https://github.com/zzfoutofspace/ATPO.

翻译：大语言模型智能体已成为解决多轮任务的有力系统，其通过交替进行内部推理与外部工具交互来实现目标。智能体强化学习作为一种关键的训练后范式，旨在进一步精炼这些能力，近来已引起广泛的研究关注。本文提出AT$^2$PO（基于树搜索的智能体回合制策略优化），这是一个面向多轮智能体强化学习的统一框架，旨在解决三个核心挑战：探索多样性有限、信用分配稀疏以及策略优化失准。AT$^2$PO引入了一种回合级树结构，该结构同时实现了用于策略探索的熵引导树扩展以及用于从稀疏结果中进行细粒度奖励传播的回合级信用分配。作为补充，我们提出了智能体回合制策略优化，这是一个回合级学习目标，它将策略更新与智能体交互的自然决策粒度对齐。AT$^2$PO与树搜索正交，可以轻松集成到任何多轮强化学习流程中。在七个基准测试上的实验表明，其性能相较于最先进的基线方法平均提升高达1.84个百分点，消融研究也验证了每个组件的有效性。我们的代码可在 https://github.com/zzfoutofspace/ATPO 获取。