Artificial Intelligence has been used to help human complete difficult tasks in complicated environments by providing optimized strategies for decision-making or replacing the manual labour. In environments including multiple agents, such as football, the most common methods to train agents are Imitation Learning and Multi-Agent Reinforcement Learning (MARL). However, the agents trained by Imitation Learning cannot outperform the expert demonstrator, which makes humans hardly get new insights from the learnt policy. Besides, MARL is prone to the credit assignment problem. In environments with sparse reward signal, this method can be inefficient. The objective of our research is to create a novel reward shaping method by embedding contextual information in reward function to solve the aforementioned challenges. We demonstrate this in the Google Research Football (GRF) environment. We quantify the contextual information extracted from game state observation and use this quantification together with original sparse reward to create the shaped reward. The experiment results in the GRF environment prove that our reward shaping method is a useful addition to state-of-the-art MARL algorithms for training agents in environments with sparse reward signal.

翻译：人工智能已用于通过提供优化决策策略或替代人工劳动，帮助人类在复杂环境中完成困难任务。在包含多个智能体的环境（如足球）中，训练智能体的最常见方法是模仿学习与多智能体强化学习。然而，通过模仿学习训练的智能体无法超越专家演示者，这使得人类难以从学习到的策略中获得新见解。此外，多智能体强化学习易受信用分配问题困扰。在奖励信号稀疏的环境中，该方法效率可能较低。本研究的目标是通过在奖励函数中嵌入上下文信息，创建一种新颖的奖励塑造方法，以解决上述挑战。我们在谷歌研究足球环境中对此进行验证。我们量化从游戏状态观测中提取的上下文信息，并将此量化结果与原始稀疏奖励结合，构建塑造奖励。在谷歌研究足球环境中的实验结果证明，我们提出的奖励塑造方法是对现有最先进多智能体强化学习算法的有效补充，适用于在奖励信号稀疏的环境中训练智能体。