One of the fundamental quests of AI is to produce agents that coordinate well with humans. This problem is challenging, especially in domains that lack high quality human behavioral data, because multi-agent reinforcement learning (RL) often converges to different equilibria from the ones that humans prefer. We propose a novel framework, instructRL, that enables humans to specify what kind of strategies they expect from their AI partners through natural language instructions. We use pretrained large language models to generate a prior policy conditioned on the human instruction and use the prior to regularize the RL objective. This leads to the RL agent converging to equilibria that are aligned with human preferences. We show that instructRL converges to human-like policies that satisfy the given instructions in a proof-of-concept environment as well as the challenging Hanabi benchmark. Finally, we show that knowing the language instruction significantly boosts human-AI coordination performance in human evaluations in Hanabi.

翻译：人工智能的基本目标之一是生成能够与人类良好协作的智能体。这一问题极具挑战性，尤其是在缺乏高质量人类行为数据的领域，因为多智能体强化学习（RL）通常会收敛到与人类偏好不同的均衡状态。我们提出了一种新颖框架——instructRL，该框架使人类能够通过自然语言指令指定其对AI伙伴的预期策略。我们利用预训练的大语言模型生成基于人类指令的先验策略，并用该先验对强化学习目标进行正则化。这促使强化学习智能体收敛到与人类偏好一致的均衡状态。我们证明，在概念验证环境以及具有挑战性的Hanabi基准测试中，instructRL能够收敛到满足给定指令的类人策略。最后，我们发现在Hanabi的人类评估中，理解语言指令能显著提升人机协作性能。