To make effective decisions in novel environments with long-horizon goals, it is crucial to engage in hierarchical reasoning across spatial and temporal scales. This entails planning abstract subgoal sequences, visually reasoning about the underlying plans, and executing actions in accordance with the devised plan through visual-motor control. We propose Compositional Foundation Models for Hierarchical Planning (HiP), a foundation model which leverages multiple expert foundation model trained on language, vision and action data individually jointly together to solve long-horizon tasks. We use a large language model to construct symbolic plans that are grounded in the environment through a large video diffusion model. Generated video plans are then grounded to visual-motor control, through an inverse dynamics model that infers actions from generated videos. To enable effective reasoning within this hierarchy, we enforce consistency between the models via iterative refinement. We illustrate the efficacy and adaptability of our approach in three different long-horizon table-top manipulation tasks.

翻译：为了在具有长期目标的新环境中做出有效决策，关键在于跨空间和时间尺度进行分层推理。这包括规划抽象的子目标序列、对底层计划进行视觉推理，以及通过视觉-运动控制执行与既定计划一致的行动。我们提出了用于分层规划的构成式基础模型（HiP），这是一种基础模型，它利用多个在语言、视觉和行动数据上单独训练的专家基础模型协同工作，以解决长期任务。我们使用大型语言模型构建符号计划，这些计划通过大型视频扩散模型在环境中得到具体落实。生成的视频计划随后通过逆动力学模型（该模型从生成的视频中推断行动）实现与视觉-运动控制的结合。为了在这一层次结构中实现有效推理，我们通过迭代精炼确保模型之间的一致性。我们在三个不同的桌面长期操控任务中展示了我们方法的有效性和适应性。