Generalizable articulated object manipulation is essential for home-assistant robots. Recent efforts focus on imitation learning from demonstrations or reinforcement learning in simulation, however, due to the prohibitive costs of real-world data collection and precise object simulation, it still remains challenging for these works to achieve broad adaptability across diverse articulated objects. Recently, many works have tried to utilize the strong in-context learning ability of Large Language Models (LLMs) to achieve generalizable robotic manipulation, but most of these researches focus on high-level task planning, sidelining low-level robotic control. In this work, building on the idea that the kinematic structure of the object determines how we can manipulate it, we propose a kinematic-aware prompting framework that prompts LLMs with kinematic knowledge of objects to generate low-level motion trajectory waypoints, supporting various object manipulation. To effectively prompt LLMs with the kinematic structure of different objects, we design a unified kinematic knowledge parser, which represents various articulated objects as a unified textual description containing kinematic joints and contact location. Building upon this unified description, a kinematic-aware planner model is proposed to generate precise 3D manipulation waypoints via a designed kinematic-aware chain-of-thoughts prompting method. Our evaluation spanned 48 instances across 16 distinct categories, revealing that our framework not only outperforms traditional methods on 8 seen categories but also shows a powerful zero-shot capability for 8 unseen articulated object categories. Moreover, the real-world experiments on 7 different object categories prove our framework's adaptability in practical scenarios. Code is released at \href{https://github.com/xwinks/LLM_articulated_object_manipulation}{here}.
翻译:可泛化铰接物体操控对家庭辅助机器人至关重要。近期研究主要聚焦于基于演示的模仿学习或仿真环境中的强化学习,但由于真实世界数据采集的高昂成本与精确物体仿真的难度,这些方法在实现广泛适配不同铰接物体方面仍面临挑战。近年来,许多工作尝试利用大语言模型(LLMs)的强上下文学习能力实现可泛化机器人操控,但多数研究集中于高层任务规划,而忽视了底层运动控制。本研究基于“物体运动学结构决定其操控方式”的核心思想,提出一种运动学感知提示框架,通过向LLMs注入物体的运动学知识生成低层运动轨迹路点,从而支持多种物体操控。为有效向LLMs提示不同物体的运动学结构,我们设计了统一运动学知识解析器,将各类铰接物体转化为包含运动学关节与接触位置的统一文本描述。基于该统一描述,我们提出运动学感知规划器模型,通过设计的运动学感知思维链提示方法生成精确的3D操控路点。评估覆盖16个不同类别的48个实例,结果表明该框架不仅在8个已知类别上优于传统方法,更在8个未见铰接物体类别上展现出强大的零样本泛化能力。此外,在7种不同物体类别上开展的真实世界实验验证了框架在实践场景中的适应性。代码已开源至\href{https://github.com/xwinks/LLM_articulated_object_manipulation}{此处}。