Reinforcement learning with group-based objectives, such as Group Relative Policy Optimization (GRPO), is a common framework for aligning large language models on complex reasoning tasks. However, standard GRPO treats each rollout trajectory as an independent flat sequence and assigns a single sequence-level advantage to all tokens, which leads to sample inefficiency and a length bias toward verbose, redundant chains of thought without improving logical depth. We introduce TreeAdv (Tree-Structured Advantage Redistribution for Group-Based RL), which makes the tree structure of group rollouts explicit for both exploration and advantage assignment. Specifically, TreeAdv builds a group of trees (a forest) based on an entropy-driven sampling method where each tree branches at high-uncertainty decisions while sharing low-uncertainty tokens across rollouts. Then, TreeAdv aggregates token-level advantages for internal tree segments by redistributing the advantages of complete rollouts (all leaf nodes), and TreeAdv can easily apply to group-based objectives such as GRPO or GSPO. Across 10 math reasoning benchmarks, TreeAdv consistently outperforms GRPO and GSPO, while using substantially fewer generated tokens under identical supervision, data, and decoding budgets.

翻译：基于分组目标的强化学习（如分组相对策略优化GRPO）是对齐大型语言模型处理复杂推理任务的常用框架。然而，标准GRPO将每个轨迹视为独立的扁平序列，并为所有词元分配单一序列级优势，这导致样本效率低下，并产生倾向于冗长冗余思维链的长度偏差，却未能提升逻辑深度。本文提出TreeAdv（面向分组强化学习的树形优势重分配），该方法在探索与优势分配中显式利用分组轨迹的树形结构。具体而言，TreeAdv基于熵驱动的采样方法构建树群（森林），其中每棵树在不确定性高的决策点进行分支，同时在轨迹间共享不确定性低的词元。随后，TreeAdv通过重分配完整轨迹（所有叶节点）的优势，聚合内部树段对应的词元级优势，该方法可轻松适用于GRPO或GSPO等分组目标。在10项数学推理基准测试中，TreeAdv在相同监督、数据和解码预算下，使用显著更少的生成词元，始终优于GRPO与GSPO方法。