Large Language Models (LLMs) have recently shown promise as high-level planners for robots when given access to a selection of low-level skills. However, it is often assumed that LLMs do not possess sufficient knowledge to be used for the low-level trajectories themselves. In this work, we address this assumption thoroughly, and investigate if an LLM (GPT-4) can directly predict a dense sequence of end-effector poses for manipulation tasks, when given access to only object detection and segmentation vision models. We designed a single, task-agnostic prompt, without any in-context examples, motion primitives, or external trajectory optimisers. Then we studied how well it can perform across 30 real-world language-based tasks, such as "open the bottle cap" and "wipe the plate with the sponge", and we investigated which design choices in this prompt are the most important. Our conclusions raise the assumed limit of LLMs for robotics, and we reveal for the first time that LLMs do indeed possess an understanding of low-level robot control sufficient for a range of common tasks, and that they can additionally detect failures and then re-plan trajectories accordingly. Videos, prompts, and code are available at: https://www.robot-learning.uk/language-models-trajectory-generators.
翻译:大型语言模型(LLMs)在获得一系列低层技能访问权限后,近期已展现出作为机器人高层规划器的潜力。然而,学界通常假设LLMs不具备足够的知识来直接生成低层运动轨迹。本研究对这一假设进行了深入探讨,并探究了当仅具备物体检测与分割视觉模型时,LLMs(GPT-4)能否直接预测操作任务中末端执行器姿态的密集序列。我们设计了一个单一、任务无关的提示词,其中不包含任何上下文示例、运动基元或外部轨迹优化器。随后,我们在30个真实世界的语言驱动任务(如“打开瓶盖”“用海绵擦拭盘子”)中系统评估其性能,并深入分析了该提示词中不同设计选择的重要性。我们的结论突破了此前对LLMs在机器人领域能力边界的假设,首次揭示出LLMs确实具备足够理解低层机器人控制的能力以完成一系列常见任务,并且能够自主检测故障并据此重新规划轨迹。视频、提示词及代码详见:https://www.robot-learning.uk/language-models-trajectory-generators。