Despite the empirical success of the deep Q network (DQN) reinforcement learning algorithm and its variants, DQN is still not well understood and it does not guarantee convergence. In this work, we show that DQN can diverge and cease to operate in realistic settings. Although there exist gradient-based convergent methods, we show that they actually have inherent problems in learning behaviour and elucidate why they often fail in practice. To overcome these problems, we propose a convergent DQN algorithm (C-DQN) by carefully modifying DQN, and we show that the algorithm is convergent and can work with large discount factors (0.9998). It learns robustly in difficult settings and can learn several difficult games in the Atari 2600 benchmark where DQN fail, within a moderate computational budget. Our codes have been publicly released and can be used to reproduce our results.

翻译：尽管深Q网络的强化学习算法及其变体取得了经验性的成功,但DQN仍然没有得到很好的理解,无法保证趋同。在这项工作中,我们表明DQN可以在现实环境中出现差异,停止在现实环境中运作。虽然存在基于梯度的趋同方法,但我们表明它们实际上在学习行为方面有内在问题,并说明了它们在实践中经常失败的原因。为了克服这些问题,我们通过仔细修改DQN提出一个趋同的DQN算法(C-DQN),我们表明,该算法在困难环境中非常活跃,可以使用大折扣系数(0.9998),在Atari 2600基准中,当DQN在其中失败时,可以在适度的计算预算范围内学习一些困难的游戏。我们的代码已经公开发布,可以用来复制我们的结果。