AI systems make decisions in physical environments through primitive actions or affordances that are accessed via API calls. While deploying AI agents in the real world involves numerous high-level actions, existing embodied simulators offer a limited set of domain-salient APIs. This naturally brings up the questions: how many primitive actions (APIs) are needed for a versatile embodied agent, and what should they look like? We explore this via a thought experiment: assuming that wikiHow tutorials cover a wide variety of human-written tasks, what is the space of APIs needed to cover these instructions? We propose a framework to iteratively induce new APIs by grounding wikiHow instruction to situated agent policies. Inspired by recent successes in large language models (LLMs) for embodied planning, we propose a few-shot prompting to steer GPT-4 to generate Pythonic programs as agent policies and bootstrap a universe of APIs by 1) reusing a seed set of APIs; and then 2) fabricate new API calls when necessary. The focus of this thought experiment is on defining these APIs rather than their executability. We apply the proposed pipeline on instructions from wikiHow tutorials. On a small fraction (0.5%) of tutorials, we induce an action space of 300+ APIs necessary for capturing the rich variety of tasks in the physical world. A detailed automatic and human analysis of the induction output reveals that the proposed pipeline enables effective reuse and creation of APIs. Moreover, a manual review revealed that existing simulators support only a small subset of the induced APIs (9 of the top 50 frequent APIs), motivating the development of action-rich embodied environments.

翻译：人工智能系统通过原始动作或可供性在物理环境中做出决策，这些动作或可供性通过API调用进行访问。尽管在现实世界中部署AI智能体涉及众多高级动作，但现有的具身模拟器仅提供有限的领域显著API。这自然引出了以下问题：一个多功能的具身智能体需要多少原始动作（API），它们又应具备何种形态？我们通过一项思想实验来探讨此问题：假设wikiHow教程涵盖了各种人类撰写的任务，那么覆盖这些指令所需的API空间是怎样的？我们提出了一个框架，通过将wikiHow指令具身化为智能体策略来迭代归纳新的API。受近期大语言模型在具身规划方面成功的启发，我们提出了一种少样本提示方法，引导GPT-4生成Python风格的程序作为智能体策略，并通过以下方式引导构建API宇宙：1）重用一组种子API；然后2）在必要时创建新的API调用。本思想实验的重点在于定义这些API，而非其可执行性。我们将所提出的流程应用于wikiHow教程中的指令。在少量（0.5%）教程中，我们归纳出300多个API的动作空间，这些API对于捕捉物理世界中丰富多样的任务是必要的。对归纳输出结果的详细自动和人工分析表明，所提出的流程能够有效实现API的重用和创建。此外，人工审查发现现有模拟器仅支持归纳出的API中的一小部分（前50个高频API中的9个），这激励了开发动作丰富的具身环境的需求。