To make effective decisions in novel environments with long-horizon goals, it is crucial to engage in hierarchical reasoning across spatial and temporal scales. This entails planning abstract subgoal sequences, visually reasoning about the underlying plans, and executing actions in accordance with the devised plan through visual-motor control. We propose Compositional Foundation Models for Hierarchical Planning (HiP), a foundation model which leverages multiple expert foundation model trained on language, vision and action data individually jointly together to solve long-horizon tasks. We use a large language model to construct symbolic plans that are grounded in the environment through a large video diffusion model. Generated video plans are then grounded to visual-motor control, through an inverse dynamics model that infers actions from generated videos. To enable effective reasoning within this hierarchy, we enforce consistency between the models via iterative refinement. We illustrate the efficacy and adaptability of our approach in three different long-horizon table-top manipulation tasks.

翻译：为了在具有长期目标的新环境中做出有效决策，必须进行跨空间和时间尺度的分层推理。这包括规划抽象子目标序列、对底层计划进行视觉推理，以及通过视觉运动控制按照制定的计划执行动作。我们提出了组合式基础模型用于分层规划（HiP），这是一种基础模型，它协同利用分别基于语言、视觉和动作数据训练的多个专家基础模型，共同解决长期任务。我们使用大型语言模型构建符号化计划，并通过大型视频扩散模型将其与环境相锚定。生成的视频计划随后通过视觉运动控制得以落地，这是经由一个逆向动力学模型实现的——该模型从生成的视频中推断动作。为了在此分层结构内实现有效推理，我们通过迭代细化确保模型之间的一致性。我们在三个不同的长期桌面操作任务中展示了该方法的效果和适应性。