Recent curriculum Reinforcement Learning (RL) has shown notable progress in solving complex tasks by proposing sequences of surrogate tasks. However, the previous approaches often face challenges when they generate curriculum goals in a high-dimensional space. Thus, they usually rely on manually specified goal spaces. To alleviate this limitation and improve the scalability of the curriculum, we propose a novel curriculum method that automatically defines the semantic goal space which contains vital information for the curriculum process, and suggests curriculum goals over it. To define the semantic goal space, our method discretizes continuous observations via vector quantized-variational autoencoders (VQ-VAE) and restores the temporal relations between the discretized observations by a graph. Concurrently, ours suggests uncertainty and temporal distance-aware curriculum goals that converges to the final goals over the automatically composed goal space. We demonstrate that the proposed method allows efficient explorations in an uninformed environment with raw goal examples only. Also, ours outperforms the state-of-the-art curriculum RL methods on data efficiency and performance, in various goal-reaching tasks even with ego-centric visual inputs.

翻译：摘要：近期课程强化学习通过提出一系列替代任务，在解决复杂任务方面取得了显著进展。然而，先前的方法在高维空间中生成课程目标时往往面临挑战，因此通常依赖手动指定的目标空间。为缓解这一局限性并提高课程的可扩展性，我们提出了一种新颖的课程方法，该方法自动定义包含课程过程关键信息的语义目标空间，并在此基础上建议课程目标。为定义语义目标空间，我们的方法通过向量量化变分自编码器（VQ-VAE）对连续观测进行离散化，并利用图结构恢复离散观测之间的时间关系。同时，该方法建议具有不确定性及时间距离感知的课程目标，在自动构建的目标空间中逐步收敛至最终目标。我们证明，所提方法仅基于原始目标示例即可在无先验信息的环境中进行高效探索。此外，在各类目标到达任务中（包括以自我为中心的视觉输入场景），本方法在数据效率和性能上均优于当前最先进的课程强化学习方法。