Reinforcement learning algorithms typically struggle in the absence of a dense, well-shaped reward function. Intrinsically motivated exploration methods address this limitation by rewarding agents for visiting novel states or transitions, but these methods offer limited benefits in large environments where most discovered novelty is irrelevant for downstream tasks. We describe a method that uses background knowledge from text corpora to shape exploration. This method, called ELLM (Exploring with LLMs) rewards an agent for achieving goals suggested by a language model prompted with a description of the agent's current state. By leveraging large-scale language model pretraining, ELLM guides agents toward human-meaningful and plausibly useful behaviors without requiring a human in the loop. We evaluate ELLM in the Crafter game environment and the Housekeep robotic simulator, showing that ELLM-trained agents have better coverage of common-sense behaviors during pretraining and usually match or improve performance on a range of downstream tasks. Code available at https://github.com/yuqingd/ellm.

翻译：强化学习算法通常在缺乏稠密且设计良好的奖励函数时表现不佳。内在动机驱动的探索方法通过奖励代理访问新状态或转换来缓解这一限制，但在大型环境中这些方法效果有限，因为大多数发现的新奇内容与下游任务无关。本文提出一种利用文本语料库中的背景知识来塑造探索过程的方法，称为ELLM（基于大型语言模型的探索）。该方法通过语言模型根据代理当前状态的描述生成目标，并奖励代理实现这些目标。借助大规模语言模型预训练的能力，ELLM引导代理向对人类有意义且可能有用的行为发展，无需人工介入。我们在Crafter游戏环境和Housekeep机器人模拟器中评估了ELLM，结果表明经过ELLM训练的代理在预训练期间能更好地覆盖常识性行为，并在大多数下游任务上达到或超越基线性能。代码开源地址：https://github.com/yuqingd/ellm。