Robot learning methods have the potential for widespread generalization across tasks, environments, and objects. However, these methods require large diverse datasets that are expensive to collect in real-world robotics settings. For robot learning to generalize, we must be able to leverage sources of data or priors beyond the robot's own experience. In this work, we posit that image-text generative models, which are pre-trained on large corpora of web-scraped data, can serve as such a data source. We show that despite these generative models being trained on largely non-robotics data, they can serve as effective ways to impart priors into the process of robot learning in a way that enables widespread generalization. In particular, we show how pre-trained generative models can serve as effective tools for semantically meaningful data augmentation. By leveraging these pre-trained models for generating appropriate "semantic" data augmentations, we propose a system GenAug that is able to significantly improve policy generalization. We apply GenAug to tabletop manipulation tasks, showing the ability to re-target behavior to novel scenarios, while only requiring marginal amounts of real-world data. We demonstrate the efficacy of this system on a number of object manipulation problems in the real world, showing a 40% improvement in generalization to novel scenes and objects.

翻译：机器人学习方法有望在任务、环境和物体之间实现广泛泛化。然而，这些方法需要大量多样化的数据集，而在真实机器人环境中收集此类数据成本高昂。为使机器人学习具备泛化能力，我们必须能够利用超出机器人自身经验的数据来源或先验知识。在本文中，我们提出：基于大规模网络爬取数据预训练的图文生成模型可充当此类数据来源。我们证明，尽管这些生成模型主要基于非机器人数据进行训练，但它们仍可有效将先验知识注入机器人学习过程，从而实现广泛泛化。具体而言，我们展示了预训练生成模型如何作为语义意义丰富的数据增强的有效工具。通过利用这些预训练模型生成合适的"语义"数据增强，我们提出了GenAug系统，该系统能够显著提升策略泛化能力。我们将GenAug应用于桌面操作任务，展示了其将行为重定向至新场景的能力，同时仅需极少量真实世界数据。我们在多项真实物体操作问题中验证了该系统的有效性，实验表明，在对新场景和新物体的泛化方面，性能提升了40%。