Deep reinforcement learning (DRL) remains challenging in tasks with sparse rewards. These sparse rewards often only indicate whether the task is partially or fully completed, meaning that many exploration actions must be performed before the agent obtains useful feedback. Hence, most existing DRL algorithms fail to learn feasible policies within a reasonable time frame. To overcome this problem, we develop an approach that exploits offline demonstration trajectories for faster and more efficient online RL in sparse reward settings. Our key insight is that by regarding offline demonstration trajectories as guidance, instead of imitating them, our method learns a policy whose state-action visitation marginal distribution matches that of offline demonstrations. Specifically, we introduce a novel trajectory distance based on maximum mean discrepancy (MMD) and formulate policy optimization as a distance-constrained optimization problem. Then, we show that this distance-constrained optimization problem can be reduced into a policy-gradient algorithm with shaped rewards learned from offline demonstrations. The proposed algorithm is evaluated on extensive discrete and continuous control tasks with sparse and deceptive rewards. The experimental results indicate that our proposed algorithm is significantly better than the baseline methods regarding diverse exploration and learning the optimal policy.

翻译：深度强化学习（DRL）在稀疏奖励任务中仍面临挑战。这些稀疏奖励通常仅指示任务是否部分或完全完成，意味着智能体需执行大量探索动作后才能获得有效反馈。因此，现有大多数DRL算法难以在合理时间内学习到可行策略。为克服此问题，我们提出一种在稀疏奖励设置下利用离线演示轨迹实现更快更高效在线强化学习的方法。核心思路在于：将离线演示轨迹视为引导信号而非直接模仿，使所学策略的状态-动作访问边际分布与离线演示匹配。具体而言，我们基于最大均值差异（MMD）引入新型轨迹距离度量，并将策略优化建模为距离约束优化问题。随后证明该距离约束优化问题可转化为基于离线演示学习塑形奖励的策略梯度算法。所提算法在包含稀疏与欺骗性奖励的广泛离散和连续控制任务上进行了评估。实验结果表明，在多样化探索与最优策略学习方面，本算法显著优于基线方法。