Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an important paradigm for unlocking reasoning capabilities in large language models, exemplified by the success of OpenAI o1 and DeepSeek-R1. Currently, Group Relative Policy Optimization (GRPO) stands as the dominant algorithm in this domain due to its stable training and critic-free efficiency. However, we argue that GRPO suffers from a structural limitation: it imposes a uniform, static trust region constraint across all samples. This design implicitly assumes signal homogeneity, a premise misaligned with the heterogeneous nature of outcome-driven learning, where advantage magnitudes and variances fluctuate significantly. Consequently, static constraints fail to fully exploit high-quality signals while insufficiently suppressing noise, often precipitating rapid entropy collapse. To address this, we propose \textbf{E}lastic \textbf{T}rust \textbf{R}egions (\textbf{ETR}), a dynamic mechanism that aligns optimization constraints with signal quality. ETR constructs a signal-aware landscape through dual-level elasticity: at the micro level, it scales clipping boundaries based on advantage magnitude to accelerate learning from high-confidence paths; at the macro level, it leverages group variance to implicitly allocate larger update budgets to tasks in the optimal learning zone. Extensive experiments on AIME and MATH benchmarks demonstrate that ETR consistently outperforms GRPO, achieving superior accuracy while effectively mitigating policy entropy degradation to ensure sustained exploration.

翻译：可验证奖励强化学习（RLVR）已成为解锁大型语言模型推理能力的重要范式，以OpenAI o1和DeepSeek-R1的成功为典型代表。目前，组相对策略优化（GRPO）因其训练稳定和无批评器的高效性，成为该领域的主导算法。然而，我们认为GRPO存在结构性局限：它对所有样本施加了统一、静态的置信域约束。这种设计隐含地假设了信号同质性，该前提与结果驱动学习的异质性本质不相符——在结果驱动学习中，优势值幅度和方差存在显著波动。因此，静态约束既无法充分利用高质量信号，又不足以抑制噪声，常常导致熵值快速崩溃。为解决这一问题，我们提出**弹性置信域（ETR）**，这是一种将优化约束与信号质量对齐的动态机制。ETR通过双层级弹性构建信号感知的优化空间：在微观层面，它根据优势值幅度动态调整截断边界，以加速从高置信度路径的学习；在宏观层面，它利用组方差隐式地为处于最优学习区间的任务分配更大的更新预算。在AIME和MATH基准上的大量实验表明，ETR始终优于GRPO，在实现更高准确率的同时，有效缓解策略熵值退化，确保持续探索能力。