Hierarchical Reinforcement Learning (HRL) frameworks like Option-Critic (OC) and Multi-updates Option Critic (MOC) have introduced significant advancements in learning reusable options. However, these methods underperform in multi-goal environments with sparse rewards, where actions must be linked to temporally distant outcomes. To address this limitation, we first propose MOC-HER, which integrates the Hindsight Experience Replay (HER) mechanism into the MOC framework. By relabeling goals from achieved outcomes, MOC-HER can solve sparse reward environments that are intractable for the original MOC. However, this approach is insufficient for object manipulation tasks, where the reward depends on the object reaching the goal rather than on the agent's direct interaction. This makes it extremely difficult for HRL agents to discover how to interact with these objects. To overcome this issue, we introduce Dual Objectives Hindsight Experience Replay (2HER), a novel extension that creates two sets of virtual goals. In addition to relabeling goals based on the object's final state (standard HER), 2HER also generates goals from the agent's effector positions, rewarding the agent for both interacting with the object and completing the task. Experimental results in robotic manipulation environments show that MOC-2HER achieves success rates of up to 90%, compared to less than 11% for both MOC and MOC-HER. These results highlight the effectiveness of our dual objective relabeling strategy in sparse reward, multi-goal tasks.

翻译：如选项批评（OC）和多更新选项批评（MOC）等分层强化学习（HRL）框架在学习可复用选项方面取得了重要进展。然而，这些方法在具有稀疏奖励的多目标环境中表现不佳，因为此类环境要求智能体将动作与时间上遥远的结果相关联。为解决这一局限，我们首先提出MOC-HER方法，将后见经验回放（HER）机制整合到MOC框架中。通过根据已实现结果重新标注目标，MOC-HER能够解决原始MOC难以处理的稀疏奖励环境。但该方法在物体操控任务中仍显不足，因为此类任务的奖励取决于物体是否到达目标位置，而非智能体与物体的直接交互。这使得HRL智能体极难发现与物体交互的有效策略。为克服这一问题，我们提出了双目标后见经验回放（2HER）这一创新扩展方法，该方法可生成两组虚拟目标。除了基于物体最终状态重新标注目标（标准HER方法）外，2HER还根据智能体执行器位置生成目标，从而对智能体与物体的交互行为和任务完成情况同时给予奖励。在机器人操控环境中的实验结果表明，MOC-2HER的成功率最高可达90%，而MOC和MOC-HER的成功率均低于11%。这些结果凸显了我们的双目标重标注策略在稀疏奖励多目标任务中的有效性。