The growing interest in language-conditioned robot manipulation aims to develop robots capable of understanding and executing complex tasks, with the objective of enabling robots to interpret language commands and manipulate objects accordingly. While language-conditioned approaches demonstrate impressive capabilities for addressing tasks in familiar environments, they encounter limitations in adapting to unfamiliar environment settings. In this study, we propose a general-purpose, language-conditioned approach that combines base skill priors and imitation learning under unstructured data to enhance the algorithm's generalization in adapting to unfamiliar environments. We assess our model's performance in both simulated and real-world environments using a zero-shot setting. In the simulated environment, the proposed approach surpasses previously reported scores for CALVIN benchmark, especially in the challenging Zero-Shot Multi-Environment setting. The average completed task length, indicating the average number of tasks the agent can continuously complete, improves more than 2.5 times compared to the state-of-the-art method HULC. In addition, we conduct a zero-shot evaluation of our policy in a real-world setting, following training exclusively in simulated environments without additional specific adaptations. In this evaluation, we set up ten tasks and achieved an average 30% improvement in our approach compared to the current state-of-the-art approach, demonstrating a high generalization capability in both simulated environments and the real world. For further details, including access to our code and videos, please refer to our supplementary materials.

翻译：随着语言条件化机器人操作领域的研究兴趣日益增长，目标在于开发能够理解并执行复杂任务的机器人，使其能够解析语言指令并据此操控物体。尽管面向语言条件化的方法在解决已知环境中的任务时展现出卓越能力，但在适应陌生环境设置时仍存在局限性。本研究提出一种通用型语言条件化方法，通过结合基础技能先验与非结构化数据下的模仿学习，增强算法在适应陌生环境时的泛化能力。我们在模拟环境与真实世界中采用零样本设置评估模型性能。在模拟环境中，所提方法超越了此前CALVIN基准测试的得分记录，尤其在具有挑战性的零样本多环境设置中表现突出。平均完成任务长度——即智能体可连续完成的任务平均数量——相比现有最优方法HULC提升了2.5倍以上。此外，我们还在真实世界环境中对策略进行零样本评估，该策略仅经模拟环境训练而无需额外特定适配。在十项任务设置的评估中，相比当前最优方法，本方法平均提升30%，展现出在模拟环境与真实世界中的强泛化能力。更多详情（含代码与视频访问方式）请参阅补充材料。