Heterogeneous Networks (HetNets) pose critical challenges for intelligent management due to the diverse user requirements and time-varying wireless conditions. These factors introduce significant decision complexity, which limits the adaptability of existing Deep Reinforcement Learning (DRL) methods. In many DRL algorithms, especially those involving value-based or actor-critic structures, the critic component plays a key role in guiding policy learning by estimating value functions. However, conventional critic models often use shallow architectures that map observations directly to scalar estimates, limiting their ability to handle multi-task complexity. In contrast, recent progress in inference-time scaling of Large Language Models (LLMs) has shown that generating intermediate reasoning steps can significantly improve decision quality. Motivated by this, we propose ReaCritic, a reasoning transformer-based critic-model scaling scheme that brings reasoning-like ability into DRL. ReaCritic performs horizontal reasoning over parallel state-action inputs and vertical reasoning through deep transformer stacks. It is compatible with a broad range of value-based and actor-critic DRL algorithms and enhances generalization in dynamic wireless environments. Extensive experiments demonstrate that ReaCritic improves convergence speed and final performance across various HetNet settings and standard OpenAI Gym control tasks. The code of ReaCritic is available at https://github.com/NICE-HKU/ReaCritic.

翻译：异构网络（HetNets）因多样化的用户需求和时变的无线条件，给智能管理带来了关键挑战。这些因素引入了显著的决策复杂性，限制了现有深度强化学习（DRL）方法的适应性。在许多DRL算法中，特别是那些涉及基于值或演员-评论家结构的算法，评论家组件通过估计价值函数在指导策略学习中起着关键作用。然而，传统的评论家模型通常采用浅层架构，将观测直接映射到标量估计，限制了其处理多任务复杂性的能力。相比之下，大型语言模型（LLMs）在推理时扩展方面的最新进展表明，生成中间推理步骤可以显著提高决策质量。受此启发，我们提出了ReaCritic，一种基于推理Transformer的评论家模型扩展方案，将类推理能力引入DRL。ReaCritic通过对并行状态-动作输入进行水平推理，并通过深度Transformer堆栈进行垂直推理。它与广泛的基于值和演员-评论家DRL算法兼容，并增强了动态无线环境中的泛化能力。大量实验表明，ReaCritic在各种HetNet设置和标准OpenAI Gym控制任务中提高了收敛速度和最终性能。ReaCritic的代码可在https://github.com/NICE-HKU/ReaCritic获取。