One of the fundamental quests of AI is to produce agents that coordinate well with humans. This problem is challenging, especially in domains that lack high quality human behavioral data, because multi-agent reinforcement learning (RL) often converges to different equilibria from the ones that humans prefer. We propose a novel framework, instructRL, that enables humans to specify what kind of strategies they expect from their AI partners through natural language instructions. We use pretrained large language models to generate a prior policy conditioned on the human instruction and use the prior to regularize the RL objective. This leads to the RL agent converging to equilibria that are aligned with human preferences. We show that instructRL converges to human-like policies that satisfy the given instructions in a proof-of-concept environment as well as the challenging Hanabi benchmark. Finally, we show that knowing the language instruction significantly boosts human-AI coordination performance in human evaluations in Hanabi.

翻译：人工智能的基本目标之一是构建能够与人类良好协同的智能体。这一问题极具挑战性，尤其是在缺乏高质量人类行为数据的领域，因为多智能体强化学习（RL）通常收敛到与人类偏好不同的均衡点。我们提出了一种新颖的框架instructRL，使人类能够通过自然语言指令指定他们期望AI伙伴采取的策略类型。我们利用预训练的大语言模型生成基于人类指令的先验策略，并以此先验正则化强化学习目标。这促使RL智能体收敛到与人类偏好一致的均衡点。我们证明，在概念验证环境以及具有挑战性的Hanabi基准测试中，instructRL能够收敛到满足给定指令的类人策略。最后，我们表明，在Hanabi环境的人类评估中，了解语言指令显著提升了人机协同表现。