Soft object manipulation tasks in domestic scenes pose a significant challenge for existing robotic skill learning techniques due to their complex dynamics and variable shape characteristics. Since learning new manipulation skills from human demonstration is an effective way for robot applications, developing prior knowledge of the representation and dynamics of soft objects is necessary. In this regard, we propose a pre-trained soft object manipulation skill learning model, namely SoftGPT, that is trained using large amounts of exploration data, consisting of a three-dimensional heterogeneous graph representation and a GPT-based dynamics model. For each downstream task, a goal-oriented policy agent is trained to predict the subsequent actions, and SoftGPT generates the consequences of these actions. Integrating these two approaches establishes a thinking process in the robot's mind that provides rollout for facilitating policy learning. Our results demonstrate that leveraging prior knowledge through this thinking process can efficiently learn various soft object manipulation skills, with the potential for direct learning from human demonstrations.

翻译：家居场景中的软体物体操作任务因其复杂的动力学特性和多变的形状特征，对现有机器人技能学习技术构成了显著挑战。由于从人类示范中学习新操作技能是机器人应用的有效途径，发展软体物体表征与动力学的先验知识至关重要。为此，我们提出一种预训练的软体物体操作技能学习模型——SoftGPT，该模型利用大量探索数据进行训练，包含三维异构图表示和基于GPT的动力学模型。针对每个下游任务，训练目标导向的策略智能体以预测后续动作，而SoftGPT则生成这些动作的结果。整合这两种方法可在机器人思维中建立思考过程，通过提供轨迹展开来促进策略学习。实验结果表明，通过这种思考过程利用先验知识，能够高效学习多种软体物体操作技能，并具备直接从人类示范中学习的潜力。