Large language models (LLMs) encode a vast amount of world knowledge acquired from massive text datasets. Recent studies have demonstrated that LLMs can assist an embodied agent in solving complex sequential decision making tasks by providing high-level instructions. However, interactions with LLMs can be time-consuming. In many practical scenarios, they require a significant amount of storage space that can only be deployed on remote cloud server nodes. Additionally, using commercial LLMs can be costly since they may charge based on usage frequency. In this paper, we explore how to enable intelligent cost-effective interactions between the agent and an LLM. We propose When2Ask, a reinforcement learning based approach that learns when it is necessary to query LLMs for high-level instructions to accomplish a target task. Experiments on MiniGrid and Habitat environments that entail planning sub-goals demonstrate that When2Ask learns to solve target tasks with only a few necessary interactions with an LLM, and significantly reduces interaction costs in testing environments compared with baseline methods. Experiment results also suggest that by learning a mediator model to interact with the LLM, the agent's performance becomes more robust against partial observability of the environment. Our code is available at https://github.com/ZJLAB-AMMI/LLM4RL.

翻译：大型语言模型（LLMs）从海量文本数据中编码了大量世界知识。近年研究表明，LLMs可通过提供高级指令辅助具身智能体解决复杂的序列决策任务。然而，与LLMs的交互可能耗时。在许多实际场景中，它们需要巨大的存储空间，仅能部署在远程云服务器节点上。此外，使用商业LLMs可能因按使用频率计费而成本高昂。本文探索如何实现智能体与LLM间经济高效的智能交互。我们提出When2Ask方法，这是一种基于强化学习的方法，可学习何时需要查询LLM以获取完成目标任务的高级指令。在涉及子目标规划的MiniGrid和Habitat环境中的实验表明，When2Ask仅需与LLM进行少量必要交互即可学习解决目标任务，且与基线方法相比显著降低了测试环境中的交互成本。实验结果还表明，通过学习中介模型与LLM交互，智能体性能对环境的局部可观测性更具鲁棒性。我们的代码已开源：https://github.com/ZJLAB-AMMI/LLM4RL。