Reasoning ability has become a defining capability of Large Language Models (LLMs), with Reinforcement Learning with Verifiable Rewards (RLVR) emerging as a key paradigm to enhance it. However, RLVR training often suffers from policy entropy collapse, where the policy becomes overly deterministic, hindering exploration and limiting reasoning performance. While entropy regularization is a common remedy, its effectiveness is highly sensitive to the fixed coefficient, making it unstable across tasks and models. In this work, we revisit entropy regularization in RLVR and argue that its potential has been largely underestimated. Our analysis shows that (i) tasks of varying difficulty demand distinct exploration intensities, and (ii) balanced exploration may require the policy entropy to be maintained within a moderate range below its initial level. Therefore, we propose Adaptive Entropy Regularization (AER)--a framework that dynamically balances exploration and exploitation via three components: difficulty-aware coefficient allocation, initial-anchored target entropy, and dynamic global coefficient adjustment. Experiments on multiple mathematical reasoning benchmarks show that AER consistently outperforms baselines, improving both reasoning accuracy and exploration capability.

翻译：推理能力已成为大型语言模型（LLM）的关键能力，而基于可验证奖励的强化学习（RLVR）已成为提升该能力的重要范式。然而，RLVR训练常面临策略熵坍塌问题——策略变得过度确定性，阻碍探索并限制推理性能。虽然熵正则化是常见解决方案，但其有效性对固定系数高度敏感，导致在不同任务和模型间不稳定。本文重新审视RLVR中的熵正则化，认为其潜力被严重低估。分析表明：(i) 不同难度的任务需要不同的探索强度，(ii) 均衡的探索可能需要将策略熵维持在初始水平以下的适度范围内。因此，我们提出自适应熵正则化（AER）框架——通过三个组件动态平衡探索与利用：难度感知系数分配、初始锚定目标熵和全局系数动态调整。多个数学推理基准上的实验表明，AER始终优于基线方法，同时提升了推理准确性和探索能力。