Soft object manipulation tasks in domestic scenes pose a significant challenge for existing robotic skill learning techniques due to their complex dynamics and variable shape characteristics. Since learning new manipulation skills from human demonstration is an effective way for robot applications, developing prior knowledge of the representation and dynamics of soft objects is necessary. In this regard, we propose a pre-trained soft object manipulation skill learning model, namely SoftGPT, that is trained using large amounts of exploration data, consisting of a three-dimensional heterogeneous graph representation and a GPT-based dynamics model. For each downstream task, a goal-oriented policy agent is trained to predict the subsequent actions, and SoftGPT generates the consequences of these actions. Integrating these two approaches establishes a thinking process in the robot's mind that provides rollout for facilitating policy learning. Our results demonstrate that leveraging prior knowledge through this thinking process can efficiently learn various soft object manipulation skills, with the potential for direct learning from human demonstrations.

翻译：在家庭场景中，软体物体操作任务因其复杂的动力学特性和可变的形状特征，对现有机器人技能学习技术构成了重大挑战。由于通过人类示范学习新操作技能是机器人应用的有效途径，因此开发软体物体表征与动力学的先验知识至关重要。为此，我们提出了一种预训练的软体物体操作技能学习模型——SoftGPT，该模型利用大量探索数据进行训练，包含三维异构图表示和基于GPT的动力学模型。针对每个下游任务，训练一个目标导向的策略智能体来预测后续动作，而SoftGPT则生成这些动作的结果。将这两种方法相结合，在机器人思维中建立了一个思考过程，为促进策略学习提供推演。我们的结果表明，通过这种思考过程利用先验知识，能够高效学习多种软体物体操作技能，并具有从人类示范中直接学习的潜力。