Building embodied agents on integrating Large Language Models (LLMs) and Reinforcement Learning (RL) have revolutionized human-AI interaction: researchers can now leverage language instructions to plan decision-making for open-ended tasks. However, existing research faces challenges in meeting the requirement of open-endedness. They typically either train LLM/RL models to adapt to a fixed counterpart, limiting exploration of novel skills and hindering the efficacy of human-AI interaction. To this end, we present OpenPAL, a co-training framework comprising two stages: (1) fine-tuning a pre-trained LLM to translate human instructions into goals for planning, and goal-conditioned training a policy for decision-making; (2) co-training to align the LLM and policy, achieving instruction open-endedness. We conducted experiments using Contra, an open-ended FPS game, demonstrating that an agent trained with OpenPAL not only comprehends arbitrary instructions but also exhibits efficient execution. These results suggest that OpenPAL holds the potential to construct open-ended embodied agents in practical scenarios.

翻译：融合大型语言模型（LLMs）与强化学习（RL）构建具身智能体已革新了人机交互方式：研究人员现可利用语言指令为开放式任务规划决策。然而，现有研究在满足开放式需求方面仍面临挑战。它们通常训练LLM/RL模型适应固定对手，这既限制了新技能的探索，又阻碍了人机交互效率的提升。为此，我们提出OpenPAL双训练框架，包含两个阶段：（1）微调预训练LLM将人类指令转化为规划目标，并基于目标条件训练策略模型进行决策；（2）联合训练对齐LLM与策略模型，实现指令的开放性。我们采用开放式第一人称射击游戏《Contra》进行实验，结果表明经OpenPAL训练的智能体不仅能理解任意指令，还能高效执行。这些结果证明OpenPAL具备在现实场景中构建开放式具身智能体的潜力。