Artificial neural networks are promising for general function approximation but challenging to train on non-independent or non-identically distributed data due to catastrophic forgetting. The experience replay buffer, a standard component in deep reinforcement learning, is often used to reduce forgetting and improve sample efficiency by storing experiences in a large buffer and using them for training later. However, a large replay buffer results in a heavy memory burden, especially for onboard and edge devices with limited memory capacities. We propose memory-efficient reinforcement learning algorithms based on the deep Q-network algorithm to alleviate this problem. Our algorithms reduce forgetting and maintain high sample efficiency by consolidating knowledge from the target Q-network to the current Q-network. Compared to baseline methods, our algorithms achieve comparable or better performance in both feature-based and image-based tasks while easing the burden of large experience replay buffers.

翻译：人工神经网络在一般函数逼近方面具有良好前景，但由于灾难性遗忘问题，在非独立同分布数据上的训练面临挑战。经验回放缓冲区作为深度强化学习中的标准组件，通常通过将经验存储在大容量缓冲区中并用于后续训练来减少遗忘并提高样本效率。然而，大的回放缓冲区会带来沉重的内存负担，尤其是对于内存容量有限的机载和边缘设备。我们提出基于深度Q网络算法的记忆高效强化学习算法来缓解这一问题。我们的算法通过将目标Q网络的知识巩固到当前Q网络，从而减少遗忘并保持高样本效率。与基线方法相比，我们的算法在基于特征和基于图像的任务中均取得了可比或更优的性能，同时减轻了大容量经验回放缓冲区的负担。