Artificial neural networks are promising for general function approximation but challenging to train on non-independent or non-identically distributed data due to catastrophic forgetting. The experience replay buffer, a standard component in deep reinforcement learning, is often used to reduce forgetting and improve sample efficiency by storing experiences in a large buffer and using them for training later. However, a large replay buffer results in a heavy memory burden, especially for onboard and edge devices with limited memory capacities. We propose memory-efficient reinforcement learning algorithms based on the deep Q-network algorithm to alleviate this problem. Our algorithms reduce forgetting and maintain high sample efficiency by consolidating knowledge from the target Q-network to the current Q-network. Compared to baseline methods, our algorithms achieve comparable or better performance in both feature-based and image-based tasks while easing the burden of large experience replay buffers.

翻译：人工神经网络在通用函数逼近方面具有广阔前景，但由于灾难性遗忘问题，在非独立或非同分布数据上进行训练颇具挑战。经验回放缓冲区作为深度强化学习的标准组件，通常通过将经验存储在大容量缓冲区中并用于后续训练来减少遗忘、提升样本效率。然而，对于内存容量有限的嵌入式及边缘设备而言，大型回放缓冲区会带来沉重的内存负担。我们提出基于深度Q网络算法的内存高效强化学习算法以缓解该问题。所提算法通过将目标Q网络的知识巩固至当前Q网络，在减少遗忘的同时保持高样本效率。与基线方法相比，我们的算法在基于特征与基于图像的任务中均能取得相当或更优性能，同时减轻了大型经验回放缓冲区的存储负担。