Off-policy actor-critic methods in reinforcement learning train a critic with temporal-difference updates and use it as a learning signal for the policy (actor). This design typically achieves higher sample efficiency than purely on-policy methods. However, critic networks tend to overestimate value estimates systematically. This is often addressed by introducing a pessimistic bias based on uncertainty estimates. Current methods employ ensembling to quantify the critic's epistemic uncertainty-uncertainty due to limited data and model ambiguity-to scale pessimistic updates. In this work, we propose a new algorithm called Stochastic Actor-Critic (STAC) that incorporates temporal (one-step) aleatoric uncertainty-uncertainty arising from stochastic transitions, rewards, and policy-induced variability in Bellman targets-to scale pessimistic bias in temporal-difference updates, rather than relying on epistemic uncertainty. STAC uses a single distributional critic network to model the temporal return uncertainty, and applies dropout to both the critic and actor networks for regularization. Our results show that pessimism based on a distributional critic alone suffices to mitigate overestimation, and naturally leads to risk-averse behavior in stochastic environments. Introducing dropout further improves training stability and performance by means of regularization. With this design, STAC achieves improved computational efficiency using a single distributional critic network.

翻译：强化学习中的离策略行动者-评论家方法通过时序差分更新训练评论家网络，并将其作为策略（行动者）的学习信号。这种设计通常比纯在策略方法具有更高的样本效率。然而，评论家网络往往系统性地高估价值估计值。现有方法通常通过引入基于不确定性估计的悲观偏差来解决此问题。当前方法采用集成学习来量化评论家的认知不确定性——即由数据有限性和模型模糊性引起的不确定性——从而调节悲观更新的程度。本文提出一种名为随机行动者-评论家（STAC）的新算法，该算法引入时序（单步）偶然不确定性——即贝尔曼目标中由随机状态转移、奖励信号以及策略诱导的变异性所产生的不确定性——来调节时序差分更新中的悲观偏差，而非依赖认知不确定性。STAC采用单一分布型评论家网络对时序回报不确定性进行建模，并对评论家与行动者网络同时应用随机失活技术以实现正则化。实验结果表明，仅基于分布型评论家的悲观机制足以缓解价值高估问题，并自然引导智能体在随机环境中表现出风险规避行为。引入随机失活技术通过正则化进一步提升了训练稳定性与算法性能。通过这种设计，STAC仅使用单一分布型评论家网络即可实现更高的计算效率。