Offline reinforcement learning (RL) is challenged by the distributional shift problem. To address this problem, existing works mainly focus on designing sophisticated policy constraints between the learned policy and the behavior policy. However, these constraints are applied equally to well-performing and inferior actions through uniform sampling, which might negatively affect the learned policy. To alleviate this issue, we propose Offline Prioritized Experience Replay (OPER), featuring a class of priority functions designed to prioritize highly-rewarding transitions, making them more frequently visited during training. Through theoretical analysis, we show that this class of priority functions induce an improved behavior policy, and when constrained to this improved policy, a policy-constrained offline RL algorithm is likely to yield a better solution. We develop two practical strategies to obtain priority weights by estimating advantages based on a fitted value network (OPER-A) or utilizing trajectory returns (OPER-R) for quick computation. OPER is a plug-and-play component for offline RL algorithms. As case studies, we evaluate OPER on five different algorithms, including BC, TD3+BC, Onestep RL, CQL, and IQL. Extensive experiments demonstrate that both OPER-A and OPER-R significantly improve the performance for all baseline methods. Codes and priority weights are availiable at https://github.com/sail-sg/OPER.

翻译：离线强化学习面临分布偏移问题的挑战。为解决该问题，现有工作主要致力于在学习策略与行为策略之间设计复杂的策略约束。然而，这些约束通过均匀采样平等地作用于表现良好和表现较差的动作，这可能会对学习策略产生负面影响。为缓解此问题，我们提出离线优先经验回放（OPER），其特征是一类旨在优先处理高奖励转移的优先级函数，使这些转移在训练过程中被更频繁地访问。通过理论分析，我们证明此类优先级函数能诱导出改进的行为策略，并且当约束于该改进策略时，基于策略约束的离线强化学习算法更可能得到更优解。我们开发了两种实用策略来获取优先级权重：基于拟合值函数估计优势的OPER-A，或利用轨迹回报进行快速计算的OPER-R。OPER是离线强化学习算法的即插即用组件。作为案例研究，我们在五种不同算法（包括BC、TD3+BC、Onestep RL、CQL和IQL）上评估OPER。大量实验表明，OPER-A和OPER-R均显著提升了所有基线方法的性能。代码和优先级权重已开源至https://github.com/sail-sg/OPER。