Large Language Models (LLMs) have emerged as a new paradigm for embodied reasoning and control, most recently by generating robot policy code that utilizes a custom library of vision and control primitive skills. However, prior arts fix their skills library and steer the LLM with carefully hand-crafted prompt engineering, limiting the agent to a stationary range of addressable tasks. In this work, we introduce LRLL, an LLM-based lifelong learning agent that continuously grows the robot skill library to tackle manipulation tasks of ever-growing complexity. LRLL achieves this with four novel contributions: 1) a soft memory module that allows dynamic storage and retrieval of past experiences to serve as context, 2) a self-guided exploration policy that proposes new tasks in simulation, 3) a skill abstractor that distills recent experiences into new library skills, and 4) a lifelong learning algorithm for enabling human users to bootstrap new skills with minimal online interaction. LRLL continuously transfers knowledge from the memory to the library, building composable, general and interpretable policies, while bypassing gradient-based optimization, thus relieving the learner from catastrophic forgetting. Empirical evaluation in a simulated tabletop environment shows that LRLL outperforms end-to-end and vanilla LLM approaches in the lifelong setup while learning skills that are transferable to the real world. Project material will become available at the webpage https://gtziafas.github.io/LRLL_project.

翻译：大型语言模型已成为具身推理与控制的新范式，其最新应用是通过生成利用定制化视觉与控制基础技能库的机器人策略代码。然而，现有研究通常固定其技能库，并依赖精心设计的人工提示工程来引导语言模型，导致智能体可处理的任务范围受限且停滞不前。本研究提出LRLL——一种基于语言模型的终身学习智能体，能够持续扩展机器人技能库以应对日益复杂的操作任务。LRLL通过四项创新实现该目标：1）软记忆模块，支持动态存储和检索过往经验作为上下文信息；2）自引导探索策略，在仿真环境中自主提出新任务；3）技能抽象器，将近期经验提炼为新的库技能；4）终身学习算法，使人类用户能以最小在线交互成本快速构建新技能。LRLL持续将记忆中的知识迁移至技能库，构建可组合、泛化性强且可解释的策略，同时规避基于梯度的优化方法，从而避免灾难性遗忘问题。在仿真桌面环境中的实证评估表明，LRLL在终身学习场景中优于端到端及原始语言模型方法，且所学技能可迁移至现实世界。项目资料将在网页 https://gtziafas.github.io/LRLL_project 发布。