Reinforcement learning for Large Language Model agents is often hindered by sparse rewards in multi-step reasoning tasks. Existing approaches like Group Relative Policy Optimization treat sampled trajectories as independent chains, assigning uniform credit to all steps in each chain and ignoring the existence of critical steps that may disproportionally impact reasoning outcome. In this paper, we propose T-STAR(Tree-structured Self-Taught Agent Rectification), a framework that recovers the latent correlated reward structure across seemingly independent trajectories. Specifically, we consolidate trajectories into a unified Cognitive Tree by identifying and merging functionally similar steps/nodes. It enables an Introspective Valuation mechanism that back-propagates trajectory-level rewards through the tree to obtain a new notion of variance-reduced relative advantage at step-level. Using the Cognitive Tree, we also develop In-Context Thought Grafting to synthesize corrective reasoning by contrasting successful and failed branches at critical divergence points/steps. Our proposed Surgical Policy Optimization then capitalizes on the rich policy gradient information concentrated at these critical points/steps through a Bradley-Terry type of surgical loss. Extensive experiments across embodied, interactive, reasoning, and planning benchmarks demonstrate that T-STAR achieves consistent improvements over strong baselines, with gains most pronounced on tasks requiring extended reasoning chains.

翻译：针对大语言模型智能体的强化学习在多步推理任务中常受稀疏奖励的制约。现有方法如分组相对策略优化将采样轨迹视为独立链式结构，对链内所有步骤赋予均等信用，忽视了可能对推理结果产生不成比例影响的关键步骤。本文提出T-STAR（树结构自教智能体修正）框架，该框架可恢复看似独立轨迹间潜在的关联奖励结构。具体而言，我们通过识别并合并功能相似的步骤/节点，将轨迹整合为统一的认知树。该机制支持内省估值方法，通过沿树反向传播轨迹级奖励，获得步骤级方差缩减的相对优势新概念。利用认知树，我们进一步开发了情境思维嫁接技术，通过对比关键分歧点/步骤处的成功与失败分支，综合生成修正性推理。提出的手术策略优化则通过Bradley-Terry式手术损失，在关键点/步骤处利用集中的丰富策略梯度信息。在具身、交互、推理与规划基准上的大量实验表明，T-STAR在强基线方法上实现持续改进，在需要长链推理的任务中提升最为显著。