Instance segmentation is a fundamental skill for many robotic applications. We propose a self-supervised method that uses grasp interactions to collect segmentation supervision for an instance segmentation model. When a robot grasps an item, the mask of that grasped item can be inferred from the images of the scene before and after the grasp. Leveraging this insight, we learn a grasp segmentation model to segment the grasped object from before and after grasp images. Such a model can segment grasped objects from thousands of grasp interactions without costly human annotation. Using the segmented grasped objects, we can "cut" objects from their original scenes and "paste" them into new scenes to generate instance supervision. We show that our grasp segmentation model provides a 5x error reduction when segmenting grasped objects compared with traditional image subtraction approaches. Combined with our "cut-and-paste" generation method, instance segmentation models trained with our method achieve better performance than a model trained with 10x the amount of labeled data. On a real robotic grasping system, our instance segmentation model reduces the rate of grasp errors by over 3x compared to an image subtraction baseline.

翻译：实例分割是许多机器人应用中的基本技能。我们提出了一种自监督方法，该方法利用抓取交互来收集实例分割模型的训练信号。当机器人抓取一个物体时，该被抓取物体的掩码可从抓取前后场景的图像中推断出来。利用这一洞察，我们训练了一个抓取分割模型，以从抓取前后图像中分割出被抓取物体。该模型能够从数千次抓取交互中分割出被抓取物体，无需昂贵的人工标注。利用分割出的被抓取物体，我们可以从原始场景中“切割”出物体，并将其“粘贴”到新场景中，从而生成实例级别的监督信号。我们证明，与传统图像减法方法相比，我们的抓取分割模型在分割被抓取物体时将误差降低了5倍。结合我们提出的“切割-粘贴”生成方法，使用本方法训练的实例分割模型，其性能优于使用10倍标注数据训练的模型。在实际机器人抓取系统中，与图像减法基线方法相比，我们的实例分割模型将抓取错误率降低了3倍以上。