The ability to wield tools was once considered exclusive to human intelligence, but it's now known that many other animals, like crows, possess this capability. Yet, robotic systems still fall short of matching biological dexterity. In this paper, we investigate the use of Large Language Models (LLMs), tool affordances, and object manoeuvrability for non-prehensile tool-based manipulation tasks. Our novel method leverages LLMs based on scene information and natural language instructions to enable symbolic task planning for tool-object manipulation. This approach allows the system to convert a human language sentence into a sequence of feasible motion functions. We have developed a novel manoeuvrability-driven controller using a new tool affordance model derived from visual feedback. This controller helps guide the robot's tool utilization and manipulation actions, even within confined areas, using a stepping incremental approach. The proposed methodology is evaluated with experiments to prove its effectiveness under various manipulation scenarios.

翻译：使用工具的能力曾被认为是人类智能的专属，但现已得知许多其他动物（如乌鸦）也具备此能力。然而，机器人系统在匹配生物灵巧性方面仍显不足。本文研究了利用大型语言模型（LLMs）、工具可供性和物体机动性来完成基于工具的非抓握式操作任务。我们的新方法基于场景信息和自然语言指令，利用LLMs实现工具-物体操作的符号化任务规划。该方法使系统能够将人类语言句子转换为一系列可行的运动函数序列。我们开发了一种新颖的机动性驱动控制器，该控制器采用了源自视觉反馈的新工具可供性模型。该控制器通过步进增量方法，即使在受限区域内，也能引导机器人的工具使用和操作动作。所提出的方法通过实验进行评估，以证明其在多种操作场景下的有效性。