Efficient exploration for an agent is challenging in reinforcement learning (RL). In this paper, a novel actor-critic framework namely virtual action actor-critic (VAAC), is proposed to address the challenge of efficient exploration in RL. This work is inspired by humans' ability to imagine the potential outcomes of their actions without actually taking them. In order to emulate this ability, VAAC introduces a new actor called virtual actor (VA), alongside the conventional actor-critic framework. Unlike the conventional actor, the VA takes the virtual action to anticipate the next state without interacting with the environment. With the virtual policy following a Gaussian distribution, the VA is trained to maximize the anticipated novelty of the subsequent state resulting from a virtual action. If any next state resulting from available actions does not exhibit high anticipated novelty, training the VA leads to an increase in the virtual policy entropy. Hence, high virtual policy entropy represents that there is no room for exploration. The proposed VAAC aims to maximize a modified Q function, which combines cumulative rewards and the negative sum of virtual policy entropy. Experimental results show that the VAAC improves the exploration performance compared to existing algorithms.

翻译：强化学习中的高效探索对智能体而言具有挑战性。本文提出了一种名为虚拟动作演员-评论家（VAAC）的新型演员-评论家框架，以应对强化学习中高效探索的难题。该研究受人类无需实际执行动作即可想象其潜在结果能力的启发。为模仿这一能力，VAAC在传统演员-评论家框架基础上引入了一个名为虚拟演员（VA）的新型角色。与传统演员不同，VA通过执行虚拟动作预测下一状态，而无需与环境交互。基于遵循高斯分布的虚拟策略，VA被训练以最大化虚拟动作所产生后续状态的预期新异性。若现有动作导致的任何下一状态均未表现出较高的预期新异性，则对VA的训练将提升虚拟策略的熵值。因此，高虚拟策略熵值表明已无探索空间。所提出的VAAC旨在最大化修正后的Q函数，该函数综合了累积奖励与虚拟策略熵的负累加值。实验结果表明，与现有算法相比，VAAC提升了探索性能。