In reinforcement learning, experience replay stores past samples for further reuse. Prioritized sampling is a promising technique to better utilize these samples. Previous criteria of prioritization include TD error, recentness and corrective feedback, which are mostly heuristically designed. In this work, we start from the regret minimization objective, and obtain an optimal prioritization strategy for Bellman update that can directly maximize the return of the policy. The theory suggests that data with higher hindsight TD error, better on-policiness and more accurate Q value should be assigned with higher weights during sampling. Thus most previous criteria only consider this strategy partially. We not only provide theoretical justifications for previous criteria, but also propose two new methods to compute the prioritization weight, namely ReMERN and ReMERT. ReMERN learns an error network, while ReMERT exploits the temporal ordering of states. Both methods outperform previous prioritized sampling algorithms in challenging RL benchmarks, including MuJoCo, Atari and Meta-World.

翻译：在强化学习中,经验重现存储了过去样本,以便进一步再利用。优先抽样是更好地利用这些样本的一个很有希望的技术。先前的优先顺序标准包括TD错误、最近和纠正反馈,这些大多是非常规设计的。在这项工作中,我们从最遗憾最小化的目标出发,为Bellman更新工作获得最佳的优先顺序战略,从而能够直接最大限度地实现政策的回报。理论表明,具有较高后视差错的数据,在取样中应分配更好的政治性和更准确的Q值,并增加重量。因此,大多数先前的标准只是部分地考虑这一战略。我们不仅为先前的标准提供了理论上的理由,而且还提出了计算优先顺序加权的两种新方法,即REMERN和REMERT。REMERN学习了一个错误网络,而RMERT则利用了各州的时间顺序。这两种方法在挑战RL基准(包括MuJoCo、Atari和Meta-World)中都比先前的优先抽样算算法更优。