Humans excel in complex long-horizon soft body manipulation tasks via flexible tool use: bread baking requires a knife to slice the dough and a rolling pin to flatten it. Often regarded as a hallmark of human cognition, tool use in autonomous robots remains limited due to challenges in understanding tool-object interactions. Here we develop an intelligent robotic system, RoboCook, which perceives, models, and manipulates elasto-plastic objects with various tools. RoboCook uses point cloud scene representations, models tool-object interactions with Graph Neural Networks (GNNs), and combines tool classification with self-supervised policy learning to devise manipulation plans. We demonstrate that from just 20 minutes of real-world interaction data per tool, a general-purpose robot arm can learn complex long-horizon soft object manipulation tasks, such as making dumplings and alphabet letter cookies. Extensive evaluations show that RoboCook substantially outperforms state-of-the-art approaches, exhibits robustness against severe external disturbances, and demonstrates adaptability to different materials.

翻译：人类擅长通过灵活使用工具完成复杂的长期软体操控任务：例如烘焙面包时，需用刀切割面团并用擀面杖将其摊平。工具使用通常被视为人类认知的标志，但受限于对工具-物体交互的理解，自主机器人在这方面的应用仍十分有限。本文提出智能机器人系统RoboCook，能够利用多种工具感知、建模并操控弹塑性物体。该系统采用点云场景表征，通过图神经网络（GNN）建模工具-物体交互，并结合工具分类与自监督策略学习制定操控方案。实验表明，仅基于每种工具20分钟的真实世界交互数据，通用机械臂即可学会复杂的长期软体操控任务，例如制作饺子和字母饼干。大量评估证明，RoboCook性能显著优于现有最优方法，能抵御严重的外部干扰，并展现出对不同材质的适应性。