Multi-task learning of deformable object manipulation is a challenging problem in robot manipulation. Most previous works address this problem in a goal-conditioned way and adapt goal images to specify different tasks, which limits the multi-task learning performance and can not generalize to new tasks. Thus, we adapt language instruction to specify deformable object manipulation tasks and propose a learning framework. We first design a unified Transformer-based architecture to understand multi-modal data and output picking and placing action. Besides, we have introduced the visible connectivity graph to tackle nonlinear dynamics and complex configuration of the deformable object. Both simulated and real experiments have demonstrated that the proposed method is effective and can generalize to unseen instructions and tasks. Compared with the state-of-the-art method, our method achieves higher success rates (87.2% on average) and has a 75.6% shorter inference time. We also demonstrate that our method performs well in real-world experiments.

翻译：可变形物体操作的多任务学习是机器人操作中的一个挑战性问题。以往的大多数工作以目标条件的方式解决该问题，并通过调整目标图像来指定不同任务，这限制了多任务学习性能且无法泛化至新任务。为此，我们采用语言指令来指定可变形物体操作任务，并提出了一种学习框架。首先设计了一个统一的基于Transformer的架构来理解多模态数据并输出抓取与放置动作。此外，引入了可见连通性图来处理可变形物体的非线性动力学与复杂构型。模拟与真实实验均表明，所提方法效果显著且能泛化至未见过的指令与任务。与当前最先进方法相比，我们的方法成功率更高（平均87.2%），推理时间缩短75.6%。同时验证了该方法在真实实验中表现优异。