We present a framework for robot skill acquisition, which 1) efficiently scale up data generation of language-labelled robot data and 2) effectively distills this data down into a robust multi-task language-conditioned visuo-motor policy. For (1), we use a large language model (LLM) to guide high-level planning, and sampling-based robot planners (e.g. motion or grasp samplers) for generating diverse and rich manipulation trajectories. To robustify this data-collection process, the LLM also infers a code-snippet for the success condition of each task, simultaneously enabling the data-collection process to detect failure and retry as well as the automatic labeling of trajectories with success/failure. For (2), we extend the diffusion policy single-task behavior-cloning approach to multi-task settings with language conditioning. Finally, we propose a new multi-task benchmark with 18 tasks across five domains to test long-horizon behavior, common-sense reasoning, tool-use, and intuitive physics. We find that our distilled policy successfully learned the robust retrying behavior in its data collection procedure, while improving absolute success rates by 33.2% on average across five domains. Code, data, and additional qualitative results are available on https://www.cs.columbia.edu/~huy/scalingup/.

翻译：我们提出一个机器人技能获取框架，该框架能够（1）高效扩展语言标注机器人数据的生成规模，（2）有效将此类数据蒸馏为鲁棒的多任务语言条件视觉-运动策略。针对（1），我们利用大型语言模型（LLM）引导高层规划，结合基于采样的机器人规划器（如运动或抓取采样器）生成多样化且丰富的操作轨迹。为增强数据采集过程的鲁棒性，LLM同时推断每个任务成功条件的代码片段，使数据采集过程能够检测故障并重试，同时自动标记轨迹的成功/失败状态。针对（2），我们将扩散策略单任务行为克隆方法扩展至多任务语言条件场景。最后，我们提出包含五个领域18项任务的新多任务基准，用于测试长时域行为、常识推理、工具使用及直观物理理解。实验表明，蒸馏策略成功继承了数据采集过程中的鲁棒重试行为，在五个领域将平均绝对成功率提升33.2%。代码、数据及补充定性结果详见https://www.cs.columbia.edu/~huy/scalingup/。