Foundation models have made significant strides in various applications, including text-to-image generation, panoptic segmentation, and natural language processing. This paper presents Instruct2Act, a framework that utilizes Large Language Models to map multi-modal instructions to sequential actions for robotic manipulation tasks. Specifically, Instruct2Act employs the LLM model to generate Python programs that constitute a comprehensive perception, planning, and action loop for robotic tasks. In the perception section, pre-defined APIs are used to access multiple foundation models where the Segment Anything Model (SAM) accurately locates candidate objects, and CLIP classifies them. In this way, the framework leverages the expertise of foundation models and robotic abilities to convert complex high-level instructions into precise policy codes. Our approach is adjustable and flexible in accommodating various instruction modalities and input types and catering to specific task demands. We validated the practicality and efficiency of our approach by assessing it on robotic tasks in different scenarios within tabletop manipulation domains. Furthermore, our zero-shot method outperformed many state-of-the-art learning-based policies in several tasks. The code for our proposed approach is available at https://github.com/OpenGVLab/Instruct2Act, serving as a robust benchmark for high-level robotic instruction tasks with assorted modality inputs.

翻译：基础模型已在多种应用中取得显著进展，包括文本到图像生成、全景分割以及自然语言处理。本文提出Instruct2Act框架，该框架利用大语言模型将多模态指令映射至机器人操作任务的连续动作。具体而言，Instruct2Act通过大语言模型生成Python程序，构成机器人任务的完整感知、规划与动作循环。在感知模块中，预定义API调用多种基础模型，其中Segment Anything Model精准定位候选目标，CLIP对其进行分类。由此，该框架融合基础模型的专业能力与机器人功能，将复杂高级指令转化为精确的策略代码。我们的方法具有可调性与灵活性，能适应多种指令模态与输入类型，并满足特定任务需求。通过在桌面操作领域不同场景下的机器人任务评估，我们验证了该方法的实用性与效率。此外，我们的零样本方法在多项任务中超越诸多基于学习的最优策略。所提方法的代码已开源（https://github.com/OpenGVLab/Instruct2Act），可作为多模态输入下高级机器人指令任务的稳健基准。