Our aim is to learn to solve long-horizon decision-making problems in complex robotics domains given low-level skills and a handful of short-horizon demonstrations containing sequences of images. To this end, we focus on learning abstract symbolic world models that facilitate zero-shot generalization to novel goals via planning. A critical component of such models is the set of symbolic predicates that define properties of and relationships between objects. In this work, we leverage pretrained vision-language models (VLMs) to propose a large set of visual predicates potentially relevant for decision-making, and to evaluate those predicates directly from camera images. At training time, we pass the proposed predicates and demonstrations into an optimization-based model-learning algorithm to obtain an abstract symbolic world model that is defined in terms of a compact subset of the proposed predicates. At test time, given a novel goal in a novel setting, we use the VLM to construct a symbolic description of the current world state, and then use a search-based planning algorithm to find a sequence of low-level skills that achieves the goal. We demonstrate empirically across experiments in both simulation and the real world that our method can generalize aggressively, applying its learned world model to solve problems with a wide variety of object types, arrangements, numbers of objects, and visual backgrounds, as well as novel goals and much longer horizons than those seen at training time.

翻译：我们的目标是在给定低级技能和少量包含图像序列的短视域演示的情况下，学习解决复杂机器人领域中的长视域决策问题。为此，我们专注于学习抽象的符号世界模型，该模型通过规划促进对新目标的零样本泛化。此类模型的一个关键组成部分是定义对象属性和对象间关系的符号谓词集合。在本工作中，我们利用预训练的视觉语言模型（VLMs）提出一组可能对决策相关的大量视觉谓词，并直接从相机图像中评估这些谓词。在训练时，我们将提出的谓词和演示输入到一个基于优化的模型学习算法中，以获得一个抽象的符号世界模型，该模型由所提出谓词的一个紧凑子集定义。在测试时，给定新环境中的新目标，我们使用VLM构建当前世界状态的符号描述，然后使用基于搜索的规划算法来寻找实现该目标的低级技能序列。我们通过仿真和真实世界的实验经验证明，我们的方法能够实现强泛化，应用其学习到的世界模型解决涉及多种对象类型、排列方式、对象数量、视觉背景的问题，以及处理训练时未见的新目标和更长的视域。