The growing interest in language-conditioned robot manipulation aims to develop robots capable of understanding and executing complex tasks, with the objective of enabling robots to interpret language commands and manipulate objects accordingly. While language-conditioned approaches demonstrate impressive capabilities for addressing tasks in familiar environments, they encounter limitations in adapting to unfamiliar environment settings. In this study, we propose a general-purpose, language-conditioned approach that combines base skill priors and imitation learning under unstructured data to enhance the algorithm's generalization in adapting to unfamiliar environments. We assess our model's performance in both simulated and real-world environments using a zero-shot setting. In the simulated environment, the proposed approach surpasses previously reported scores for CALVIN benchmark, especially in the challenging Zero-Shot Multi-Environment setting. The average completed task length, indicating the average number of tasks the agent can continuously complete, improves more than 2.5 times compared to the state-of-the-art method HULC. In addition, we conduct a zero-shot evaluation of our policy in a real-world setting, following training exclusively in simulated environments without additional specific adaptations. In this evaluation, we set up ten tasks and achieved an average 30% improvement in our approach compared to the current state-of-the-art approach, demonstrating a high generalization capability in both simulated environments and the real world. For further details, including access to our code and videos, please refer to https://hk-zh.github.io/spil/

翻译：语言条件机器人操作的日益关注旨在开发能够理解并执行复杂任务的机器人，其目标是使机器人能够解读语言指令并据此操作物体。尽管语言条件方法在处理熟悉环境中的任务时展现出令人印象深刻的能力，但在适应陌生环境设置方面仍存在局限性。在本研究中，我们提出了一种通用的语言条件方法，结合了基础技能先验与基于非结构化数据的模仿学习，以增强算法在适应陌生环境时的泛化能力。我们在仿真环境与现实环境中采用零样本设置评估了模型性能。在仿真环境中，所提方法在CALVIN基准测试中超越了此前报告的成绩，尤其是在具有挑战性的零样本多环境设置下。平均完成任务长度（即智能体能连续完成的平均任务数）相比当前最先进方法HULC提升了2.5倍以上。此外，我们还在现实环境中对策略进行了零样本评估，该策略仅基于仿真环境训练，未做额外特定调整。在此评估中，我们设置了十项任务，相较于当前最先进方法实现了平均30%的性能提升，表明该方法在仿真环境与现实世界中均具有高度泛化能力。更多细节（包括代码与视频访问），请参见https://hk-zh.github.io/spil/