The challenges inherent to long-horizon tasks in robotics persist due to the typical inefficient exploration and sparse rewards in traditional reinforcement learning approaches. To alleviate these challenges, we introduce a novel algorithm, Variational Autoencoder-based Subgoal Inference (VAESI), to accomplish long-horizon tasks through a divide-and-conquer manner. VAESI consists of three components: a Variational Autoencoder (VAE)-based Subgoal Generator, a Hindsight Sampler, and a Value Selector. The VAE-based Subgoal Generator draws inspiration from the human capacity to infer subgoals and reason about the final goal in the context of these subgoals. It is composed of an explicit encoder model, engineered to generate subgoals, and an implicit decoder model, designed to enhance the quality of the generated subgoals by predicting the final goal. Additionally, the Hindsight Sampler selects valid subgoals from an offline dataset to enhance the feasibility of the generated subgoals. The Value Selector utilizes the value function in reinforcement learning to filter the optimal subgoals from subgoal candidates. To validate our method, we conduct several long-horizon tasks in both simulation and the real world, including one locomotion task and three manipulation tasks. The obtained quantitative and qualitative data indicate that our approach achieves promising performance compared to other baseline methods. These experimental results can be seen in the website \url{https://sites.google.com/view/vaesi/home}.

翻译：针对长期任务中传统强化学习方法普遍存在的探索效率低下和奖励稀疏问题，本文提出了一种新型算法——基于变分自编码器的子目标推理（VAESI），通过"分而治之"策略完成长期任务。VAESI包含三个核心组件：基于变分自编码器（VAE）的子目标生成器、事后采样器和价值选择器。其中，VAE子目标生成器借鉴人类推断子目标并基于子目标推理最终目标的能力，由显式编码器模型（用于生成子目标）和隐式解码器模型（通过预测最终目标提升子目标生成质量）构成。事后采样器从离线数据集中筛选有效子目标以增强生成子目标的可行性，价值选择器则利用强化学习中的价值函数从候选子目标中筛选最优子目标。为验证该方法，我们在仿真环境和真实场景中开展了包含1个移动任务和3个操作任务的长期实验。定性与定量结果表明，与基线方法相比，本方法取得了显著性能提升。完整实验数据详见项目网站：\url{https://sites.google.com/view/vaesi/home}。