Offline reinforcement learning suffers from the out-of-distribution issue and extrapolation error. Most policy constraint methods regularize the density of the trained policy towards the behavior policy, which is too restrictive in most cases. We propose Supported Trust Region optimization (STR) which performs trust region policy optimization with the policy constrained within the support of the behavior policy, enjoying the less restrictive support constraint. We show that, when assuming no approximation and sampling error, STR guarantees strict policy improvement until convergence to the optimal support-constrained policy in the dataset. Further with both errors incorporated, STR still guarantees safe policy improvement for each step. Empirical results validate the theory of STR and demonstrate its state-of-the-art performance on MuJoCo locomotion domains and much more challenging AntMaze domains.

翻译：离线强化学习面临分布外推断和外推误差问题。大多数策略约束方法将训练策略的密度向行为策略进行正则化，这在多数情况下过于严格。我们提出支持信任区域优化方法（STR），该方法在执行信任区域策略优化的同时，将策略约束在行为策略的支持集内，从而享受更宽松的支持约束。我们证明，在假设不存在近似和采样误差的情况下，STR能保证策略严格改进直至收敛到数据集中的最优支持约束策略。当同时考虑这两种误差时，STR仍能保证每一步的安全策略改进。实验结果验证了STR的理论分析，并表明其在MuJoCo运动控制领域以及更具挑战性的AntMaze领域均达到了当前最优性能。