Text-based games are a popular testbed for language-based reinforcement learning (RL). In previous work, deep Q-learning is commonly used as the learning agent. Q-learning algorithms are challenging to apply to complex real-world domains due to, for example, their instability in training. Therefore, in this paper, we adapt the soft-actor-critic (SAC) algorithm to the text-based environment. To deal with sparse extrinsic rewards from the environment, we combine it with a potential-based reward shaping technique to provide more informative (dense) reward signals to the RL agent. We apply our method to play difficult text-based games. The SAC method achieves higher scores than the Q-learning methods on many games with only half the number of training steps. This shows that it is well-suited for text-based games. Moreover, we show that the reward shaping technique helps the agent to learn the policy faster and achieve higher scores. In particular, we consider a dynamically learned value function as a potential function for shaping the learner's original sparse reward signals.

翻译：文本类游戏是基于语言进行强化学习（RL）的流行测试平台。以往工作通常采用深度Q学习作为学习智能体。此类Q学习算法由于训练不稳定等问题，难以应用于复杂现实领域。因此，本文针对文本环境改编了软演员-评论家（SAC）算法。为应对环境中稀疏的外部奖励，我们将其与基于势能的奖励塑形技术相结合，为RL智能体提供更具信息性（密集）的奖励信号。我们将该方法应用于攻克高难度文本类游戏。在众多游戏中，SAC方法仅需Q学习方法一半的训练步数即可获得更优得分，表明其非常适合文本类游戏。此外，研究表明奖励塑形技术可帮助智能体更快学习策略并取得更高得分。特别地，我们采用动态学习的值函数作为势函数，对学习者原始稀疏奖励信号进行塑形。