In image-based robot manipulation tasks with large observation and action spaces, reinforcement learning struggles with low sample efficiency, slow training speed, and uncertain convergence. As an alternative, large pre-trained foundation models have shown promise in robotic manipulation, particularly in zero-shot and few-shot applications. However, using these models directly is unreliable due to limited reasoning capabilities and challenges in understanding physical and spatial contexts. This paper introduces ExploRLLM, a novel approach that leverages the inductive bias of foundation models (e.g. Large Language Models) to guide exploration in reinforcement learning. We also exploit these foundation models to reformulate the action and observation spaces to enhance the training efficiency in reinforcement learning. Our experiments demonstrate that guided exploration enables much quicker convergence than training without it. Additionally, we validate that ExploRLLM outperforms vanilla foundation model baselines and that the policy trained in simulation can be applied in real-world settings without additional training.

翻译：在具有大规模观测空间和动作空间的基于图像的机器人操作任务中，强化学习面临样本效率低、训练速度慢及收敛性不确定等挑战。作为替代方案，大规模预训练基础模型在机器人操作领域展现出潜力，尤其在零样本和少样本应用中表现突出。然而，由于这类模型推理能力有限且难以理解物理与空间上下文，直接使用存在不可靠性。本文提出一种名为ExploRLLM的新方法，利用基础模型（如大语言模型）的归纳偏置来引导强化学习中的探索，并进一步利用这些基础模型重构动作与观测空间以提升强化学习训练效率。实验表明，引导式探索比无引导训练能实现更快速的收敛。此外，我们验证了ExploRLLM优于原始基础模型基线，且仿真环境中训练的策略可直接迁移至真实场景，无需额外训练。