Reinforcement learning (RL) with sparse and deceptive rewards is challenging because non-zero rewards are rarely obtained. Hence, the gradient calculated by the agent can be stochastic and without valid information. Recent studies that utilize memory buffers of previous experiences can lead to a more efficient learning process. However, existing methods often require these experiences to be successful and may overly exploit them, which can cause the agent to adopt suboptimal behaviors. This paper develops an approach that uses diverse past trajectories for faster and more efficient online RL, even if these trajectories are suboptimal or not highly rewarded. The proposed algorithm combines a policy improvement step with an additional exploration step using offline demonstration data. The main contribution of this paper is that by regarding diverse past trajectories as guidance, instead of imitating them, our method directs its policy to follow and expand past trajectories while still being able to learn without rewards and approach optimality. Furthermore, a novel diversity measurement is introduced to maintain the team's diversity and regulate exploration. The proposed algorithm is evaluated on discrete and continuous control tasks with sparse and deceptive rewards. Compared with the existing RL methods, the experimental results indicate that our proposed algorithm is significantly better than the baseline methods regarding diverse exploration and avoiding local optima.

翻译：具有稀疏和欺骗性奖励的强化学习（RL）极具挑战性，因为非零奖励极难获得。因此，智能体计算的梯度可能具有随机性且缺乏有效信息。近期利用先前经验记忆缓冲区的研究虽能提升学习效率，但现有方法往往要求这些经验必须成功，且可能过度利用它们，导致智能体采用次优行为。本文提出一种方法，能够利用多样化历史轨迹实现更快更高效的在线强化学习，即便这些轨迹是次优或奖励不高。该算法将策略改进步骤与使用离线演示数据的额外探索步骤相结合。本文的主要贡献在于：将多样化历史轨迹视为引导而非简单模仿，使策略能够遵循并扩展历史轨迹，同时仍能在无奖励条件下学习并趋近最优性。此外，我们引入新型多样性度量以维持团队多样性并调节探索。在具有稀疏和欺骗性奖励的离散与连续控制任务上的评估表明，与现有强化学习方法相比，本算法在多样化探索和避免局部最优方面显著优于基线方法。