Foundation models have made significant strides in various applications, including text-to-image generation, panoptic segmentation, and natural language processing. This paper presents Instruct2Act, a framework that utilizes Large Language Models to map multi-modal instructions to sequential actions for robotic manipulation tasks. Specifically, Instruct2Act employs the LLM model to generate Python programs that constitute a comprehensive perception, planning, and action loop for robotic tasks. In the perception section, pre-defined APIs are used to access multiple foundation models where the Segment Anything Model (SAM) accurately locates candidate objects, and CLIP classifies them. In this way, the framework leverages the expertise of foundation models and robotic abilities to convert complex high-level instructions into precise policy codes. Our approach is adjustable and flexible in accommodating various instruction modalities and input types and catering to specific task demands. We validated the practicality and efficiency of our approach by assessing it on robotic tasks in different scenarios within tabletop manipulation domains. Furthermore, our zero-shot method outperformed many state-of-the-art learning-based policies in several tasks. The code for our proposed approach is available at https://github.com/OpenGVLab/Instruct2Act, serving as a robust benchmark for high-level robotic instruction tasks with assorted modality inputs.

翻译：基础模型在文本生成图像、全景分割及自然语言处理等多项应用中取得了显著进展。本文提出Instruct2Act框架，该框架利用大语言模型将多模态指令映射为机器人操控任务的序列化动作。具体而言，Instruct2Act借助LLM模型生成Python程序，构成涵盖感知、规划与执行的完整机器人任务循环。在感知环节，通过预定义API调用多个基础模型：Segment Anything Model (SAM) 精准定位候选物体，CLIP对其进行分类。该框架由此融合基础模型与机器人能力的专长，将复杂高层指令转化为精确策略代码。我们的方法具有高度的可调整性与灵活性，能适配多种指令模态与输入类型，并满足特定任务需求。通过评估桌面操控场景中不同情境下的机器人任务，我们验证了该方法的实用性与效率。此外，在多项任务中，我们的零样本方法性能超越了许多当前最优的基于学习的策略。所提方法的代码已开源至https://github.com/OpenGVLab/Instruct2Act，可作为高层级多模态输入机器人指令任务的稳健基准。