Language is compositional; an instruction can express multiple relation constraints to hold among objects in a scene that a robot is tasked to rearrange. Our focus in this work is an instructable scene-rearranging framework that generalizes to longer instructions and to spatial concept compositions never seen at training time. We propose to represent language-instructed spatial concepts with energy functions over relative object arrangements. A language parser maps instructions to corresponding energy functions and an open-vocabulary visual-language model grounds their arguments to relevant objects in the scene. We generate goal scene configurations by gradient descent on the sum of energy functions, one per language predicate in the instruction. Local vision-based policies then re-locate objects to the inferred goal locations. We test our model on established instruction-guided manipulation benchmarks, as well as benchmarks of compositional instructions we introduce. We show our model can execute highly compositional instructions zero-shot in simulation and in the real world. It outperforms language-to-action reactive policies and Large Language Model planners by a large margin, especially for long instructions that involve compositions of multiple spatial concepts. Simulation and real-world robot execution videos, as well as our code and datasets are publicly available on our website: https://ebmplanner.github.io.

翻译：语言具有组合性；一条指令可以表达多个关系约束，这些约束作用于机器人需重排场景中的物体。本研究聚焦于一种可指令驱动的场景重排框架，该框架能泛化至更长指令及训练时未见过的空间概念组合。我们提出用相对物体排列的能量函数来表示语言指令所指定的空间概念：语言解析器将指令映射至对应的能量函数，而开放词汇的视觉语言模型则将其参数关联至场景中的相关物体。我们通过对每个指令谓词对应的能量函数之和进行梯度下降，生成目标场景配置，然后由基于局部视觉的策略将物体重新定位至推断出的目标位置。我们在已有的指令引导操作基准以及新引入的组合指令基准上测试了模型。结果表明，该模型能在仿真和现实世界中零样本执行高度组合的指令，尤其在涉及多个空间概念组合的长指令任务中，其性能远超语言-动作反应式策略和大语言模型规划器。仿真与真实机器人执行视频、代码及数据集均已在我们的网站公开：https://ebmplanner.github.io。