Prioritized Experience Replay (PER) is a technical means of deep reinforcement learning by selecting experience samples with more knowledge quantity to improve the training rate of neural network. However, the non-uniform sampling used in PER inevitably shifts the state-action space distribution and brings the estimation error of Q-value function. In this paper, an Attention Loss Adjusted Prioritized (ALAP) Experience Replay algorithm is proposed, which integrates the improved Self-Attention network with Double-Sampling mechanism to fit the hyperparameter that can regulate the importance sampling weights to eliminate the estimation error caused by PER. In order to verify the effectiveness and generality of the algorithm, the ALAP is tested with value-function based, policy-gradient based and multi-agent reinforcement learning algorithms in OPENAI gym, and comparison studies verify the advantage and efficiency of the proposed training framework.

翻译：优先经验回放（PER）是一种深度强化学习的技术手段，通过选择知识量更丰富的经验样本来提升神经网络的训练效率。然而，PER中使用的非均匀采样不可避免地会改变状态-动作空间的分布，并引发Q值函数的估计误差。本文提出了一种基于注意力损失调整的优先经验回放（ALAP）算法，该算法将改进的自注意力网络与双采样机制相结合，以拟合可调节重要性采样权重的超参数，从而消除PER带来的估计误差。为验证该算法的有效性和通用性，我们在OpenAI Gym中将其与基于价值函数、基于策略梯度以及多智能体强化学习算法进行了测试，对比研究证明了所提训练框架的优越性和效率。