Foundation models pre-trained on web-scale data are shown to encapsulate extensive world knowledge beneficial for robotic manipulation in the form of task planning. However, the actual physical implementation of these plans often relies on task-specific learning methods, which require significant data collection and struggle with generalizability. In this work, we introduce Robotic Manipulation through Spatial Constraints of Parts (CoPa), a novel framework that leverages the common sense knowledge embedded within foundation models to generate a sequence of 6-DoF end-effector poses for open-world robotic manipulation. Specifically, we decompose the manipulation process into two phases: task-oriented grasping and task-aware motion planning. In the task-oriented grasping phase, we employ foundation vision-language models (VLMs) to select the object's grasping part through a novel coarse-to-fine grounding mechanism. During the task-aware motion planning phase, VLMs are utilized again to identify the spatial geometry constraints of task-relevant object parts, which are then used to derive post-grasp poses. We also demonstrate how CoPa can be seamlessly integrated with existing robotic planning algorithms to accomplish complex, long-horizon tasks. Our comprehensive real-world experiments show that CoPa possesses a fine-grained physical understanding of scenes, capable of handling open-set instructions and objects with minimal prompt engineering and without additional training. Project page: https://copa-2024.github.io/

翻译：在互联网规模数据上预训练的基础模型被证明能够封装丰富的世界知识，这些知识以任务规划的形式有益于机器人操控。然而，这些规划的实际物理实现通常依赖于特定任务的学习方法，这需要大量的数据收集且难以泛化。在本工作中，我们提出基于零部件空间约束的机器人操控（CoPa），一种新颖的框架，它利用基础模型中嵌入的常识知识，为开放世界中的机器人操控生成一系列六自由度末端执行器姿态。具体而言，我们将操控过程分解为两个阶段：面向任务的抓取和任务感知的运动规划。在面向任务的抓取阶段，我们采用基础视觉语言模型（VLMs），通过一种新颖的由粗到精的定位机制来选择物体的抓取部位。在任务感知的运动规划阶段，我们再次使用VLMs来识别与任务相关零部件空间几何约束，进而推导出抓取后的姿态。我们还展示了CoPa如何能与现有的机器人规划算法无缝集成，以完成复杂的、长期的任务。我们全面的真实世界实验表明，CoPa具备对场景的细粒度物理理解能力，能够在仅需少量提示工程且无需额外训练的情况下处理开放集指令与物体。项目页面：https://copa-2024.github.io/