Goal-Conditioned Reinforcement Learning (GCRL) can enable agents to spontaneously set diverse goals to learn a set of skills. Despite the excellent works proposed in various fields, reaching distant goals in temporally extended tasks remains a challenge for GCRL. Current works tackled this problem by leveraging planning algorithms to plan intermediate subgoals to augment GCRL. Their methods need two crucial requirements: (i) a state representation space to search valid subgoals, and (ii) a distance function to measure the reachability of subgoals. However, they struggle to scale to high-dimensional state space due to their non-compact representations. Moreover, they cannot collect high-quality training data through standard GC policies, which results in an inaccurate distance function. Both affect the efficiency and performance of planning and policy learning. In the paper, we propose a goal-conditioned RL algorithm combined with Disentanglement-based Reachability Planning (REPlan) to solve temporally extended tasks. In REPlan, a Disentangled Representation Module (DRM) is proposed to learn compact representations which disentangle robot poses and object positions from high-dimensional observations in a self-supervised manner. A simple REachability discrimination Module (REM) is also designed to determine the temporal distance of subgoals. Moreover, REM computes intrinsic bonuses to encourage the collection of novel states for training. We evaluate our REPlan in three vision-based simulation tasks and one real-world task. The experiments demonstrate that our REPlan significantly outperforms the prior state-of-the-art methods in solving temporally extended tasks.

翻译：目标条件强化学习（GCRL）能够使智能体自主设定多样化的目标，从而学习一系列技能。尽管各领域已提出诸多优秀工作，但在时间延展任务中抵达远期目标仍是GCRL面临的挑战。现有研究通过引入规划算法生成中间子目标来增强GCRL，这类方法需满足两个关键条件：（i）用于搜索有效子目标的状态表征空间；（ii）用于度量子目标可达性的距离函数。然而，由于非紧致表征的限制，现有方法难以扩展到高维状态空间。此外，它们无法通过标准GC策略收集高质量训练数据，导致距离函数不准确。这两方面因素共同影响了规划与策略学习的效率与性能。本文提出一种结合解耦可达性规划（REPlan）的目标条件强化学习算法来解决时间延展任务。在REPlan中，我们设计了解耦表征模块（DRM），通过自监督方式从高维观测中学习解耦机器人位姿与物体位置的紧致表征。同时构建了简单有效的可达性判别模块（REM），用于判断子目标的时间距离。此外，REM计算内在奖励以激励收集新颖状态用于训练。我们在三个基于视觉的仿真任务和一个真实场景任务中评估了REPlan，实验表明该方法在解决时间延展任务时显著优于现有最先进方法。