Deep reinforcement learning algorithms often use two networks for value function optimization: an online network, and a target network that tracks the online network with some delay. Using two separate networks enables the agent to hedge against issues that arise when performing bootstrapping. In this paper we endow two popular deep reinforcement learning algorithms, namely DQN and Rainbow, with updates that incentivize the online network to remain in the proximity of the target network. This improves the robustness of deep reinforcement learning in presence of noisy updates. The resultant agents, called DQN Pro and Rainbow Pro, exhibit significant performance improvements over their original counterparts on the Atari benchmark demonstrating the effectiveness of this simple idea in deep reinforcement learning. The code for our paper is available here: Github.com/amazon-research/fast-rl-with-slow-updates.

翻译：深度强化学习算法通常使用两个网络进行价值函数优化：一个在线网络，以及一个以某种延迟跟踪在线网络的目标网络。使用两个独立的网络使智能体能够规避执行自举时出现的问题。在本文中，我们为两种流行的深度强化学习算法（即 DQN 和 Rainbow）赋予更新机制，激励在线网络保持在目标网络的邻近区域内。这提升了深度强化学习在存在噪声更新时的鲁棒性。由此产生的智能体称为 DQN Pro 和 Rainbow Pro，在 Atari 基准测试上相较于原始版本展现出显著的性能提升，这证明了这一简单思想在深度强化学习中的有效性。本文的代码可在以下地址获取：Github.com/amazon-research/fast-rl-with-slow-updates。