Most offline reinforcement learning (RL) algorithms return a target policy maximizing a trade-off between (1) the expected performance gain over the behavior policy that collected the dataset, and (2) the risk stemming from the out-of-distribution-ness of the induced state-action occupancy. It follows that the performance of the target policy is strongly related to the performance of the behavior policy and, thus, the trajectory return distribution of the dataset. We show that in mixed datasets consisting of mostly low-return trajectories and minor high-return trajectories, state-of-the-art offline RL algorithms are overly restrained by low-return trajectories and fail to exploit high-performing trajectories to the fullest. To overcome this issue, we show that, in deterministic MDPs with stochastic initial states, the dataset sampling can be re-weighted to induce an artificial dataset whose behavior policy has a higher return. This re-weighted sampling strategy may be combined with any offline RL algorithm. We further analyze that the opportunity for performance improvement over the behavior policy correlates with the positive-sided variance of the returns of the trajectories in the dataset. We empirically show that while CQL, IQL, and TD3+BC achieve only a part of this potential policy improvement, these same algorithms combined with our reweighted sampling strategy fully exploit the dataset. Furthermore, we empirically demonstrate that, despite its theoretical limitation, the approach may still be efficient in stochastic environments. The code is available at https://github.com/Improbable-AI/harness-offline-rl.

翻译：大多数离线强化学习算法返回一个目标策略，该策略在以下两者之间进行权衡优化：(1) 相对于收集数据集的行为策略的预期性能提升，以及(2) 由诱导的状态-动作占据分布超出分布范围所带来的风险。因此，目标策略的性能与行为策略的性能以及数据集的轨迹回报分布密切相关。我们表明，在主要由低回报轨迹和少量高回报轨迹组成的混合数据集中，现有最优的离线强化学习算法会受到低回报轨迹的过度约束，无法充分利用高表现轨迹。为解决此问题，我们证明，在具有随机初始状态的确定性马尔可夫决策过程中，可以通过对数据集采样进行重新加权，从而生成一个行为策略具有更高回报的人工数据集。这种重新加权采样策略可与任何离线强化学习算法结合使用。我们进一步分析发现，相对于行为策略的性能提升机会与数据集中轨迹回报的正侧方差相关。实验表明，虽然CQL、IQL和TD3+BC仅实现了这种潜在策略改进的一部分，但这些算法与我们的重新加权采样策略结合后，能够充分利用数据集。此外，尽管存在理论局限性，我们通过实验证明该方法在随机环境中仍可能有效。代码已开源在https://github.com/Improbable-AI/harness-offline-rl。