Reinforcement Learning from Verifier Rewards (RLVR) has emerged as a widely used approach for post-training large language models on reasoning tasks, with group-based methods such as GRPO and its variants gaining broad adoption. These methods rely on group-relative advantage estimation to avoid learned critics, yet its theoretical properties remain poorly understood. In this work, we characterize conditional estimation bias in group-relative advantages among non-degenerate groups. For mean-centered advantages under binary rewards, our analysis shows negative conditional mean estimation errors for hard prompts and positive errors for easy prompts. At the response level, correct answers to hard prompts are underweighted, while incorrect answers to easy prompts receive insufficient penalties. To address this issue, we propose History-Aware Adaptive Difficulty Weighting (HA-DW), an adaptive reweighting scheme that adjusts advantage estimates based on an evolving difficulty anchor and training dynamics. Both theoretical analysis and experiments on five mathematical reasoning benchmarks demonstrate that HA-DW consistently improves performance when integrated into GRPO and its variants. Our results suggest that correcting biased advantage estimation is critical for robust and efficient RLVR training.

翻译：暂无翻译